# Psyche State Sync

Real-time state synchronization for Psyche AI virtual assistants.

## Overview

`psyche-state-sync` provides comprehensive state management and synchronization
for AI virtual assistant sessions, including:

- **Avatar State**: Position, rotation, expression (52 FACS blendshapes), gaze,
  animations, and lip sync visemes
- **Voice State**: Speech recognition, text-to-speech, viseme timing, and
  audio-visual synchronization
- **Session State**: Lifecycle, conversation tracking, emotional state, and tool
  execution
- **Conferencing State**: Multi-participant sessions with turn-taking, screen
  sharing, and shared data

## Installation

```bash
# Basic installation
poetry add psyche-state-sync

# With Redis support for broadcasting
poetry add psyche-state-sync[redis]
```

## Quick Start

### Avatar State Management

```python
from psyche_state_sync import AvatarStateManager, AvatarAnimationState

# Create manager
manager = AvatarStateManager(
    session_id="sess-1",
    avatar_id="avatar-1"
)
await manager.initialize()

# Update expression (FACS blendshapes)
await manager.set_expression(
    mouth_smile_left=0.8,
    mouth_smile_right=0.8,
    eye_squint_left=0.3,
    eye_squint_right=0.3,
)

# Set animation
await manager.set_animation("wave", speed=1.5)

# Update gaze direction
await manager.set_gaze(
    target_x=0.5,
    target_y=0.3,
    tracking_face=True,
    attention_level=0.9,
)

# Set lip sync viseme
from psyche_state_sync import Viseme
await manager.set_viseme(Viseme.AA, weight=0.8)

# Subscribe to changes
manager.on_expression_change(lambda event: print(f"Expression: {event}"))
manager.on_animation_change(lambda event: print(f"Animation: {event}"))
```

### Voice Pipeline State

```python
from psyche_state_sync import VoiceStateManager, Viseme, VisemeFrame

manager = VoiceStateManager(
    session_id="sess-1",
    persona_id="persona-1"
)
await manager.initialize()

# Start listening for speech
await manager.start_listening()

# Update transcript (from STT)
await manager.update_transcript(
    "Hello, how are you?",
    confidence=0.95,
    is_final=True
)

# Start TTS synthesis
await manager.start_synthesis(
    text="I'm doing great!",
    model="eleven_labs_v2",
    duration_ms=2500
)

# Queue visemes for lip sync
visemes = [
    VisemeFrame(viseme=Viseme.AA, weight=1.0, timestamp_ms=0, duration_ms=100),
    VisemeFrame(viseme=Viseme.I, weight=1.0, timestamp_ms=100, duration_ms=80),
    VisemeFrame(viseme=Viseme.SIL, weight=0.5, timestamp_ms=180, duration_ms=50),
]
await manager.queue_visemes(visemes)
await manager.start_viseme_playback()

# Track detected emotion in voice
await manager.set_detected_emotion("happy", confidence=0.85)
```

### Session State Management

```python
from psyche_state_sync import SessionStateManager, ConversationPhase

manager = SessionStateManager(
    session_id="sess-1",
    user_id="user-1",
    persona_id="persona-1",
    channel="web"
)
await manager.initialize()

# Start session
await manager.start_session()

# Track conversation
await manager.set_conversation_phase(ConversationPhase.MAIN)
await manager.increment_turn(is_user=True)
await manager.add_topic("customer_support")
await manager.add_intent("complaint")
await manager.set_key_fact("order_number", "12345")

# Track emotional state
await manager.update_emotional_state(
    primary_emotion="frustrated",
    confidence=0.75,
    engagement=0.8,
    satisfaction=0.4,
    frustration=0.6
)

# Execute tools
await manager.start_tool_call("call-1", "lookup_order")
await manager.complete_tool_call("call-1", result={"status": "shipped"})

# End session
await manager.end_session(reason="resolved")
```

### Conferencing State

```python
from psyche_state_sync import (
    ConferencingStateManager,
    ParticipantRole,
    SpeakingMode,
    TurnState
)

manager = ConferencingStateManager(
    session_id="sess-1",
    room_id="room-1"
)
await manager.initialize()

# Configure room
await manager.configure_room(
    name="Support Call",
    max_participants=10,
    recording_enabled=True
)
await manager.start_room()

# Add participants
await manager.add_participant(
    participant_id="p-1",
    display_name="John",
    role=ParticipantRole.HOST,
    can_speak=True,
    can_share_screen=True,
    can_moderate=True
)

await manager.add_participant(
    participant_id="ai-1",
    display_name="AI Assistant",
    role=ParticipantRole.AI_ASSISTANT
)
await manager.set_ai_participant("ai-1")

# Manage turn-taking
await manager.set_speaking_mode(SpeakingMode.RAISE_HAND)
await manager.raise_hand("p-2")
await manager.set_current_speaker("p-2")

# Screen sharing
await manager.start_screen_share(
    sharer_id="p-1",
    remote_control_enabled=True,
    annotations_enabled=True
)

# Shared data
await manager.set_shared_data("document_url", "https://...")
```

## Broadcasting with Redis Pub/Sub

Integrate with `psyche-cache` for distributed state synchronization:

```python
from psyche_cache import create_pubsub_client
from psyche_state_sync import (
    AvatarStateManager,
    BroadcasterManager,
    BroadcastConfig
)

# Create pub/sub client
pubsub = await create_pubsub_client()

# Create broadcaster with configuration
config = BroadcastConfig(
    broadcast_full_state=True,
    broadcast_deltas=True,
    min_broadcast_interval_ms=16,  # ~60fps
    heartbeat_interval_ms=5000,
    source_id="server-1"
)

broadcaster = BroadcasterManager(
    pubsub=pubsub,
    session_id="sess-1",
    config=config
)
await broadcaster.start()

# Create state manager
avatar_manager = AvatarStateManager(
    session_id="sess-1",
    avatar_id="avatar-1"
)
await avatar_manager.initialize()

# Connect manager to broadcaster
avatar_manager.on_broadcast(
    lambda state: broadcaster.broadcast_avatar(state, avatar_manager.version)
)

# Subscribe to incoming updates
broadcaster.avatar.on_message(
    lambda msg: print(f"Received avatar update: v{msg.version}")
)

# Request sync from other broadcasters
await broadcaster.avatar.request_sync()
```

## State Types

### Avatar Expression (FACS Blendshapes)

The avatar expression model includes 52 FACS-based blendshapes:

- **Brow**: inner_up, down_left/right, outer_up_left/right
- **Eye**: look_up/down/in/out, blink, squint, wide (left/right)
- **Cheek**: puff, squint_left/right
- **Nose**: sneer_left/right
- **Jaw**: open, forward, left/right
- **Mouth**: funnel, pucker, left/right, roll_upper/lower, smile, frown, dimple,
  etc.
- **Tongue**: out

### Visemes (Lip Sync)

Standard viseme set for speech animation:

| Viseme | Phonemes  | Description   |
| ------ | --------- | ------------- |
| SIL    | -         | Silence       |
| PP     | p, b, m   | Bilabial      |
| FF     | f, v      | Labiodental   |
| TH     | th        | Dental        |
| DD     | t, d, n   | Alveolar      |
| KK     | k, g, ng  | Velar         |
| CH     | ch, j, sh | Postalveolar  |
| SS     | s, z      | Sibilant      |
| NN     | n, l      | Nasal/Lateral |
| RR     | r         | Retroflex     |
| AA     | a         | Open vowel    |
| E      | e         | Mid vowel     |
| I      | i         | Close front   |
| O      | o         | Mid back      |
| U      | u         | Close back    |

### Voice Synchronization

The voice sync status tracks audio-visual alignment:

```python
VoiceSyncStatus(
    is_synced=True,
    offset_ms=15,  # Visual ahead by 15ms
    quality=0.95,  # Sync quality
    drift_rate=0.1,  # ms/s drift
    last_correction_ms=1705500000000
)
```

## Conflict Resolution

State synchronization supports multiple conflict resolution strategies:

```python
from psyche_state_sync import ConflictResolution, create_synchronizer

sync = create_synchronizer(
    state_class=AvatarState,
    initial_state=initial,
    conflict_strategy=ConflictResolution.LAST_WRITE_WINS  # Default
    # or: ConflictResolution.FIRST_WRITE_WINS
    # or: ConflictResolution.MERGE
)
```

## Checkpointing

Save and restore state snapshots:

```python
# Get snapshot
snapshot = manager.get_snapshot()

# Save to storage
await storage.save("session-state", snapshot)

# Restore from snapshot
loaded = await storage.load("session-state")
await manager.restore_snapshot(loaded)
```

## Event System

All state managers emit typed events:

```python
from psyche_state_sync import StateEventType

# Avatar events
manager.on_state_change(handler)  # AVATAR_STATE_CHANGED
manager.on_expression_change(handler)  # AVATAR_EXPRESSION_CHANGED
manager.on_animation_change(handler)  # AVATAR_ANIMATION_STARTED/ENDED

# Voice events
manager.on_speaking_started(handler)  # VOICE_SPEAKING_STARTED
manager.on_speaking_ended(handler)  # VOICE_SPEAKING_ENDED
manager.on_listening_started(handler)  # VOICE_LISTENING_STARTED
manager.on_transcript(handler)  # VOICE_TRANSCRIPT_INTERIM/FINAL
manager.on_viseme_change(handler)  # VOICE_VISEME_CHANGED

# Session events
manager.on_session_started(handler)  # SESSION_STARTED
manager.on_session_ended(handler)  # SESSION_ENDED
manager.on_session_paused(handler)  # SESSION_PAUSED
manager.on_session_resumed(handler)  # SESSION_RESUMED

# Conferencing events
manager.on_participant_joined(handler)  # PARTICIPANT_JOINED
manager.on_participant_left(handler)  # PARTICIPANT_LEFT
manager.on_speaker_changed(handler)  # SPEAKER_CHANGED
manager.on_screen_share_started(handler)  # SCREEN_SHARE_STARTED
```

## Dependencies

- `pydantic` >= 2.5.0 - State model validation
- `psyche-cache` - Caching and pub/sub (local dependency)
- `redis` (optional) - For distributed broadcasting

## API Reference

### State Managers

| Manager                    | Purpose                | Key Methods                                                          |
| -------------------------- | ---------------------- | -------------------------------------------------------------------- |
| `AvatarStateManager`       | Avatar rendering state | `set_expression()`, `set_animation()`, `set_viseme()`, `set_gaze()`  |
| `VoiceStateManager`        | Voice pipeline state   | `start_listening()`, `update_transcript()`, `queue_visemes()`        |
| `SessionStateManager`      | Session lifecycle      | `start_session()`, `update_emotional_state()`, `start_tool_call()`   |
| `ConferencingStateManager` | Multi-party sessions   | `add_participant()`, `set_current_speaker()`, `start_screen_share()` |

### Broadcasters

| Broadcaster                    | Purpose                              |
| ------------------------------ | ------------------------------------ |
| `StateBroadcaster[T]`          | Generic state broadcaster            |
| `AvatarStateBroadcaster`       | Avatar-specific broadcasting         |
| `VoiceStateBroadcaster`        | Voice-specific broadcasting          |
| `SessionStateBroadcaster`      | Session-specific broadcasting        |
| `ConferencingStateBroadcaster` | Conferencing-specific broadcasting   |
| `BroadcasterManager`           | Unified manager for all broadcasters |

## License

Proprietary - Oshun Platform