Real-time state synchronization for Psyche AI virtual assistants.
Overview#
psyche-state-sync provides comprehensive state management and synchronization
for AI virtual assistant sessions, including:
- Avatar State: Position, rotation, expression (52 FACS blendshapes), gaze, animations, and lip sync visemes
- Voice State: Speech recognition, text-to-speech, viseme timing, and audio-visual synchronization
- Session State: Lifecycle, conversation tracking, emotional state, and tool execution
- Conferencing State: Multi-participant sessions with turn-taking, screen sharing, and shared data
Installation#
bash
# Basic installation
poetry add psyche-state-sync
# With Redis support for broadcasting
poetry add psyche-state-sync[redis]
Quick Start#
Avatar State Management#
python
from psyche_state_sync import AvatarStateManager, AvatarAnimationState
# Create manager
manager = AvatarStateManager(
session_id="sess-1",
avatar_id="avatar-1"
)
await manager.initialize()
# Update expression (FACS blendshapes)
await manager.set_expression(
mouth_smile_left=0.8,
mouth_smile_right=0.8,
eye_squint_left=0.3,
eye_squint_right=0.3,
)
# Set animation
await manager.set_animation("wave", speed=1.5)
# Update gaze direction
await manager.set_gaze(
target_x=0.5,
target_y=0.3,
tracking_face=True,
attention_level=0.9,
)
# Set lip sync viseme
from psyche_state_sync import Viseme
await manager.set_viseme(Viseme.AA, weight=0.8)
# Subscribe to changes
manager.on_expression_change(lambda event: print(f"Expression: {event}"))
manager.on_animation_change(lambda event: print(f"Animation: {event}"))
Voice Pipeline State#
python
from psyche_state_sync import VoiceStateManager, Viseme, VisemeFrame
manager = VoiceStateManager(
session_id="sess-1",
persona_id="persona-1"
)
await manager.initialize()
# Start listening for speech
await manager.start_listening()
# Update transcript (from STT)
await manager.update_transcript(
"Hello, how are you?",
confidence=0.95,
is_final=True
)
# Start TTS synthesis
await manager.start_synthesis(
text="I'm doing great!",
model="eleven_labs_v2",
duration_ms=2500
)
# Queue visemes for lip sync
visemes = [
VisemeFrame(viseme=Viseme.AA, weight=1.0, timestamp_ms=0, duration_ms=100),
VisemeFrame(viseme=Viseme.I, weight=1.0, timestamp_ms=100, duration_ms=80),
VisemeFrame(viseme=Viseme.SIL, weight=0.5, timestamp_ms=180, duration_ms=50),
]
await manager.queue_visemes(visemes)
await manager.start_viseme_playback()
# Track detected emotion in voice
await manager.set_detected_emotion("happy", confidence=0.85)
Session State Management#
python
from psyche_state_sync import SessionStateManager, ConversationPhase
manager = SessionStateManager(
session_id="sess-1",
user_id="user-1",
persona_id="persona-1",
channel="web"
)
await manager.initialize()
# Start session
await manager.start_session()
# Track conversation
await manager.set_conversation_phase(ConversationPhase.MAIN)
await manager.increment_turn(is_user=True)
await manager.add_topic("customer_support")
await manager.add_intent("complaint")
await manager.set_key_fact("order_number", "12345")
# Track emotional state
await manager.update_emotional_state(
primary_emotion="frustrated",
confidence=0.75,
engagement=0.8,
satisfaction=0.4,
frustration=0.6
)
# Execute tools
await manager.start_tool_call("call-1", "lookup_order")
await manager.complete_tool_call("call-1", result={"status": "shipped"})
# End session
await manager.end_session(reason="resolved")
Conferencing State#
python
from psyche_state_sync import (
ConferencingStateManager,
ParticipantRole,
SpeakingMode,
TurnState
)
manager = ConferencingStateManager(
session_id="sess-1",
room_id="room-1"
)
await manager.initialize()
# Configure room
await manager.configure_room(
name="Support Call",
max_participants=10,
recording_enabled=True
)
await manager.start_room()
# Add participants
await manager.add_participant(
participant_id="p-1",
display_name="John",
role=ParticipantRole.HOST,
can_speak=True,
can_share_screen=True,
can_moderate=True
)
await manager.add_participant(
participant_id="ai-1",
display_name="AI Assistant",
role=ParticipantRole.AI_ASSISTANT
)
await manager.set_ai_participant("ai-1")
# Manage turn-taking
await manager.set_speaking_mode(SpeakingMode.RAISE_HAND)
await manager.raise_hand("p-2")
await manager.set_current_speaker("p-2")
# Screen sharing
await manager.start_screen_share(
sharer_id="p-1",
remote_control_enabled=True,
annotations_enabled=True
)
# Shared data
await manager.set_shared_data("document_url", "https://...")
Broadcasting with Redis Pub/Sub#
Integrate with psyche-cache for distributed state synchronization:
python
from psyche_cache import create_pubsub_client
from psyche_state_sync import (
AvatarStateManager,
BroadcasterManager,
BroadcastConfig
)
# Create pub/sub client
pubsub = await create_pubsub_client()
# Create broadcaster with configuration
config = BroadcastConfig(
broadcast_full_state=True,
broadcast_deltas=True,
min_broadcast_interval_ms=16, # ~60fps
heartbeat_interval_ms=5000,
source_id="server-1"
)
broadcaster = BroadcasterManager(
pubsub=pubsub,
session_id="sess-1",
config=config
)
await broadcaster.start()
# Create state manager
avatar_manager = AvatarStateManager(
session_id="sess-1",
avatar_id="avatar-1"
)
await avatar_manager.initialize()
# Connect manager to broadcaster
avatar_manager.on_broadcast(
lambda state: broadcaster.broadcast_avatar(state, avatar_manager.version)
)
# Subscribe to incoming updates
broadcaster.avatar.on_message(
lambda msg: print(f"Received avatar update: v{msg.version}")
)
# Request sync from other broadcasters
await broadcaster.avatar.request_sync()
State Types#
Avatar Expression (FACS Blendshapes)#
The avatar expression model includes 52 FACS-based blendshapes:
- Brow: inner_up, down_left/right, outer_up_left/right
- Eye: look_up/down/in/out, blink, squint, wide (left/right)
- Cheek: puff, squint_left/right
- Nose: sneer_left/right
- Jaw: open, forward, left/right
- Mouth: funnel, pucker, left/right, roll_upper/lower, smile, frown, dimple, etc.
- Tongue: out
Visemes (Lip Sync)#
Standard viseme set for speech animation:
| Viseme | Phonemes | Description |
|---|---|---|
| SIL | - | Silence |
| PP | p, b, m | Bilabial |
| FF | f, v | Labiodental |
| TH | th | Dental |
| DD | t, d, n | Alveolar |
| KK | k, g, ng | Velar |
| CH | ch, j, sh | Postalveolar |
| SS | s, z | Sibilant |
| NN | n, l | Nasal/Lateral |
| RR | r | Retroflex |
| AA | a | Open vowel |
| E | e | Mid vowel |
| I | i | Close front |
| O | o | Mid back |
| U | u | Close back |
Voice Synchronization#
The voice sync status tracks audio-visual alignment:
python
VoiceSyncStatus(
is_synced=True,
offset_ms=15, # Visual ahead by 15ms
quality=0.95, # Sync quality
drift_rate=0.1, # ms/s drift
last_correction_ms=1705500000000
)
Conflict Resolution#
State synchronization supports multiple conflict resolution strategies:
python
from psyche_state_sync import ConflictResolution, create_synchronizer
sync = create_synchronizer(
state_class=AvatarState,
initial_state=initial,
conflict_strategy=ConflictResolution.LAST_WRITE_WINS # Default
# or: ConflictResolution.FIRST_WRITE_WINS
# or: ConflictResolution.MERGE
)
Checkpointing#
Save and restore state snapshots:
python
# Get snapshot
snapshot = manager.get_snapshot()
# Save to storage
await storage.save("session-state", snapshot)
# Restore from snapshot
loaded = await storage.load("session-state")
await manager.restore_snapshot(loaded)
Event System#
All state managers emit typed events:
python
from psyche_state_sync import StateEventType
# Avatar events
manager.on_state_change(handler) # AVATAR_STATE_CHANGED
manager.on_expression_change(handler) # AVATAR_EXPRESSION_CHANGED
manager.on_animation_change(handler) # AVATAR_ANIMATION_STARTED/ENDED
# Voice events
manager.on_speaking_started(handler) # VOICE_SPEAKING_STARTED
manager.on_speaking_ended(handler) # VOICE_SPEAKING_ENDED
manager.on_listening_started(handler) # VOICE_LISTENING_STARTED
manager.on_transcript(handler) # VOICE_TRANSCRIPT_INTERIM/FINAL
manager.on_viseme_change(handler) # VOICE_VISEME_CHANGED
# Session events
manager.on_session_started(handler) # SESSION_STARTED
manager.on_session_ended(handler) # SESSION_ENDED
manager.on_session_paused(handler) # SESSION_PAUSED
manager.on_session_resumed(handler) # SESSION_RESUMED
# Conferencing events
manager.on_participant_joined(handler) # PARTICIPANT_JOINED
manager.on_participant_left(handler) # PARTICIPANT_LEFT
manager.on_speaker_changed(handler) # SPEAKER_CHANGED
manager.on_screen_share_started(handler) # SCREEN_SHARE_STARTED
Dependencies#
pydantic>= 2.5.0 - State model validationpsyche-cache- Caching and pub/sub (local dependency)redis(optional) - For distributed broadcasting
API Reference#
State Managers#
| Manager | Purpose | Key Methods |
|---|---|---|
AvatarStateManager |
Avatar rendering state | set_expression(), set_animation(), set_viseme(), set_gaze() |
VoiceStateManager |
Voice pipeline state | start_listening(), update_transcript(), queue_visemes() |
SessionStateManager |
Session lifecycle | start_session(), update_emotional_state(), start_tool_call() |
ConferencingStateManager |
Multi-party sessions | add_participant(), set_current_speaker(), start_screen_share() |
Broadcasters#
| Broadcaster | Purpose |
|---|---|
StateBroadcaster[T] |
Generic state broadcaster |
AvatarStateBroadcaster |
Avatar-specific broadcasting |
VoiceStateBroadcaster |
Voice-specific broadcasting |
SessionStateBroadcaster |
Session-specific broadcasting |
ConferencingStateBroadcaster |
Conferencing-specific broadcasting |
BroadcasterManager |
Unified manager for all broadcasters |
License#
Proprietary - Oshun Platform