Eve library

Psyche State Sync

Other

Revision 1

On this page

Real-time state synchronization for Psyche AI virtual assistants.

Overview#

psyche-state-sync provides comprehensive state management and synchronization for AI virtual assistant sessions, including:

  • Avatar State: Position, rotation, expression (52 FACS blendshapes), gaze, animations, and lip sync visemes
  • Voice State: Speech recognition, text-to-speech, viseme timing, and audio-visual synchronization
  • Session State: Lifecycle, conversation tracking, emotional state, and tool execution
  • Conferencing State: Multi-participant sessions with turn-taking, screen sharing, and shared data

Installation#

bash
# Basic installation
poetry add psyche-state-sync

# With Redis support for broadcasting
poetry add psyche-state-sync[redis]

Quick Start#

Avatar State Management#

python
from psyche_state_sync import AvatarStateManager, AvatarAnimationState

# Create manager
manager = AvatarStateManager(
    session_id="sess-1",
    avatar_id="avatar-1"
)
await manager.initialize()

# Update expression (FACS blendshapes)
await manager.set_expression(
    mouth_smile_left=0.8,
    mouth_smile_right=0.8,
    eye_squint_left=0.3,
    eye_squint_right=0.3,
)

# Set animation
await manager.set_animation("wave", speed=1.5)

# Update gaze direction
await manager.set_gaze(
    target_x=0.5,
    target_y=0.3,
    tracking_face=True,
    attention_level=0.9,
)

# Set lip sync viseme
from psyche_state_sync import Viseme
await manager.set_viseme(Viseme.AA, weight=0.8)

# Subscribe to changes
manager.on_expression_change(lambda event: print(f"Expression: {event}"))
manager.on_animation_change(lambda event: print(f"Animation: {event}"))

Voice Pipeline State#

python
from psyche_state_sync import VoiceStateManager, Viseme, VisemeFrame

manager = VoiceStateManager(
    session_id="sess-1",
    persona_id="persona-1"
)
await manager.initialize()

# Start listening for speech
await manager.start_listening()

# Update transcript (from STT)
await manager.update_transcript(
    "Hello, how are you?",
    confidence=0.95,
    is_final=True
)

# Start TTS synthesis
await manager.start_synthesis(
    text="I'm doing great!",
    model="eleven_labs_v2",
    duration_ms=2500
)

# Queue visemes for lip sync
visemes = [
    VisemeFrame(viseme=Viseme.AA, weight=1.0, timestamp_ms=0, duration_ms=100),
    VisemeFrame(viseme=Viseme.I, weight=1.0, timestamp_ms=100, duration_ms=80),
    VisemeFrame(viseme=Viseme.SIL, weight=0.5, timestamp_ms=180, duration_ms=50),
]
await manager.queue_visemes(visemes)
await manager.start_viseme_playback()

# Track detected emotion in voice
await manager.set_detected_emotion("happy", confidence=0.85)

Session State Management#

python
from psyche_state_sync import SessionStateManager, ConversationPhase

manager = SessionStateManager(
    session_id="sess-1",
    user_id="user-1",
    persona_id="persona-1",
    channel="web"
)
await manager.initialize()

# Start session
await manager.start_session()

# Track conversation
await manager.set_conversation_phase(ConversationPhase.MAIN)
await manager.increment_turn(is_user=True)
await manager.add_topic("customer_support")
await manager.add_intent("complaint")
await manager.set_key_fact("order_number", "12345")

# Track emotional state
await manager.update_emotional_state(
    primary_emotion="frustrated",
    confidence=0.75,
    engagement=0.8,
    satisfaction=0.4,
    frustration=0.6
)

# Execute tools
await manager.start_tool_call("call-1", "lookup_order")
await manager.complete_tool_call("call-1", result={"status": "shipped"})

# End session
await manager.end_session(reason="resolved")

Conferencing State#

python
from psyche_state_sync import (
    ConferencingStateManager,
    ParticipantRole,
    SpeakingMode,
    TurnState
)

manager = ConferencingStateManager(
    session_id="sess-1",
    room_id="room-1"
)
await manager.initialize()

# Configure room
await manager.configure_room(
    name="Support Call",
    max_participants=10,
    recording_enabled=True
)
await manager.start_room()

# Add participants
await manager.add_participant(
    participant_id="p-1",
    display_name="John",
    role=ParticipantRole.HOST,
    can_speak=True,
    can_share_screen=True,
    can_moderate=True
)

await manager.add_participant(
    participant_id="ai-1",
    display_name="AI Assistant",
    role=ParticipantRole.AI_ASSISTANT
)
await manager.set_ai_participant("ai-1")

# Manage turn-taking
await manager.set_speaking_mode(SpeakingMode.RAISE_HAND)
await manager.raise_hand("p-2")
await manager.set_current_speaker("p-2")

# Screen sharing
await manager.start_screen_share(
    sharer_id="p-1",
    remote_control_enabled=True,
    annotations_enabled=True
)

# Shared data
await manager.set_shared_data("document_url", "https://...")

Broadcasting with Redis Pub/Sub#

Integrate with psyche-cache for distributed state synchronization:

python
from psyche_cache import create_pubsub_client
from psyche_state_sync import (
    AvatarStateManager,
    BroadcasterManager,
    BroadcastConfig
)

# Create pub/sub client
pubsub = await create_pubsub_client()

# Create broadcaster with configuration
config = BroadcastConfig(
    broadcast_full_state=True,
    broadcast_deltas=True,
    min_broadcast_interval_ms=16,  # ~60fps
    heartbeat_interval_ms=5000,
    source_id="server-1"
)

broadcaster = BroadcasterManager(
    pubsub=pubsub,
    session_id="sess-1",
    config=config
)
await broadcaster.start()

# Create state manager
avatar_manager = AvatarStateManager(
    session_id="sess-1",
    avatar_id="avatar-1"
)
await avatar_manager.initialize()

# Connect manager to broadcaster
avatar_manager.on_broadcast(
    lambda state: broadcaster.broadcast_avatar(state, avatar_manager.version)
)

# Subscribe to incoming updates
broadcaster.avatar.on_message(
    lambda msg: print(f"Received avatar update: v{msg.version}")
)

# Request sync from other broadcasters
await broadcaster.avatar.request_sync()

State Types#

Avatar Expression (FACS Blendshapes)#

The avatar expression model includes 52 FACS-based blendshapes:

  • Brow: inner_up, down_left/right, outer_up_left/right
  • Eye: look_up/down/in/out, blink, squint, wide (left/right)
  • Cheek: puff, squint_left/right
  • Nose: sneer_left/right
  • Jaw: open, forward, left/right
  • Mouth: funnel, pucker, left/right, roll_upper/lower, smile, frown, dimple, etc.
  • Tongue: out

Visemes (Lip Sync)#

Standard viseme set for speech animation:

Viseme Phonemes Description
SIL - Silence
PP p, b, m Bilabial
FF f, v Labiodental
TH th Dental
DD t, d, n Alveolar
KK k, g, ng Velar
CH ch, j, sh Postalveolar
SS s, z Sibilant
NN n, l Nasal/Lateral
RR r Retroflex
AA a Open vowel
E e Mid vowel
I i Close front
O o Mid back
U u Close back

Voice Synchronization#

The voice sync status tracks audio-visual alignment:

python
VoiceSyncStatus(
    is_synced=True,
    offset_ms=15,  # Visual ahead by 15ms
    quality=0.95,  # Sync quality
    drift_rate=0.1,  # ms/s drift
    last_correction_ms=1705500000000
)

Conflict Resolution#

State synchronization supports multiple conflict resolution strategies:

python
from psyche_state_sync import ConflictResolution, create_synchronizer

sync = create_synchronizer(
    state_class=AvatarState,
    initial_state=initial,
    conflict_strategy=ConflictResolution.LAST_WRITE_WINS  # Default
    # or: ConflictResolution.FIRST_WRITE_WINS
    # or: ConflictResolution.MERGE
)

Checkpointing#

Save and restore state snapshots:

python
# Get snapshot
snapshot = manager.get_snapshot()

# Save to storage
await storage.save("session-state", snapshot)

# Restore from snapshot
loaded = await storage.load("session-state")
await manager.restore_snapshot(loaded)

Event System#

All state managers emit typed events:

python
from psyche_state_sync import StateEventType

# Avatar events
manager.on_state_change(handler)  # AVATAR_STATE_CHANGED
manager.on_expression_change(handler)  # AVATAR_EXPRESSION_CHANGED
manager.on_animation_change(handler)  # AVATAR_ANIMATION_STARTED/ENDED

# Voice events
manager.on_speaking_started(handler)  # VOICE_SPEAKING_STARTED
manager.on_speaking_ended(handler)  # VOICE_SPEAKING_ENDED
manager.on_listening_started(handler)  # VOICE_LISTENING_STARTED
manager.on_transcript(handler)  # VOICE_TRANSCRIPT_INTERIM/FINAL
manager.on_viseme_change(handler)  # VOICE_VISEME_CHANGED

# Session events
manager.on_session_started(handler)  # SESSION_STARTED
manager.on_session_ended(handler)  # SESSION_ENDED
manager.on_session_paused(handler)  # SESSION_PAUSED
manager.on_session_resumed(handler)  # SESSION_RESUMED

# Conferencing events
manager.on_participant_joined(handler)  # PARTICIPANT_JOINED
manager.on_participant_left(handler)  # PARTICIPANT_LEFT
manager.on_speaker_changed(handler)  # SPEAKER_CHANGED
manager.on_screen_share_started(handler)  # SCREEN_SHARE_STARTED

Dependencies#

  • pydantic >= 2.5.0 - State model validation
  • psyche-cache - Caching and pub/sub (local dependency)
  • redis (optional) - For distributed broadcasting

API Reference#

State Managers#

Manager Purpose Key Methods
AvatarStateManager Avatar rendering state set_expression(), set_animation(), set_viseme(), set_gaze()
VoiceStateManager Voice pipeline state start_listening(), update_transcript(), queue_visemes()
SessionStateManager Session lifecycle start_session(), update_emotional_state(), start_tool_call()
ConferencingStateManager Multi-party sessions add_participant(), set_current_speaker(), start_screen_share()

Broadcasters#

Broadcaster Purpose
StateBroadcaster[T] Generic state broadcaster
AvatarStateBroadcaster Avatar-specific broadcasting
VoiceStateBroadcaster Voice-specific broadcasting
SessionStateBroadcaster Session-specific broadcasting
ConferencingStateBroadcaster Conferencing-specific broadcasting
BroadcasterManager Unified manager for all broadcasters

License#

Proprietary - Oshun Platform