# Psyche Voice Engine

Real-time voice processing engine for TTS, STT, and conversational AI.

Part of the Psyche AI Virtual Assistant Platform.

## Overview

The Voice Engine provides comprehensive voice processing capabilities:

- **Text-to-Speech (TTS)** - 8 provider integrations with streaming support
- **Speech-to-Text (STT)** - 5 provider integrations with real-time
  transcription
- **Viseme Generation** - MPEG-4 compliant lip-sync with coarticulation
- **Conversation Management** - LLM integration and context handling
- **Audio Processing** - VAD, noise suppression, echo cancellation

## Features

- **Multi-Provider TTS**: ElevenLabs, OpenAI, Cartesia, Deepgram Aura, F5-TTS,
  Piper, Orpheus
- **Multi-Provider STT**: Deepgram, Whisper, AssemblyAI, Google Cloud Speech
- **Real-Time Streaming**: WebSocket-based audio streaming
- **Lip-Sync Generation**: 19-viseme MPEG-4 standard with blend shapes
- **Natural Dialogue**: Turn-taking, endpointing, interruption handling
- **Failover Resilience**: Circuit breaker pattern with health monitoring
- **GPU Acceleration**: CUDA-optimized for PyTorch models

## Architecture

```
voice_engine/
├── audio/              # Audio input processing
│   ├── capture.py      # Audio capture
│   ├── vad.py          # Voice activity detection
│   └── quality.py      # Noise suppression, AGC
├── tts/                # Text-to-speech
│   ├── providers/      # Provider implementations
│   ├── streaming.py    # Real-time streaming
│   └── cache.py        # Response caching
├── stt/                # Speech-to-text
│   ├── providers/      # Provider implementations
│   └── streaming.py    # Real-time transcription
├── viseme/             # Lip-sync generation
│   ├── generator.py    # Viseme pipeline
│   └── coarticulation.py
├── phoneme/            # Phoneme processing
│   └── g2p.py          # Grapheme-to-phoneme
├── conversation/       # Dialogue management
│   ├── llm.py          # LLM integration
│   └── context.py      # Context management
├── turn_taking/        # Natural dialogue flow
│   ├── endpointing.py  # Utterance detection
│   └── interruption.py # Interruption handling
├── sync/               # A/V synchronization
│   └── timing.py       # Drift correction
├── failover/           # Provider reliability
│   └── circuit_breaker.py
└── transcript/         # Transcript processing
    ├── streaming.py    # Real-time updates
    └── enhancement.py  # Punctuation, entities
```

## Quick Start

### Installation

```bash
# Using Nx
nx install psyche-voice-engine

# With all provider SDKs
nx install-all-providers psyche-voice-engine

# Or directly with Poetry
cd apps/psyche/voice-engine
poetry install
poetry install --extras all-providers
```

### Environment Variables

```bash
# TTS Providers
ELEVENLABS_API_KEY=
OPENAI_API_KEY=
CARTESIA_API_KEY=

# STT Providers
DEEPGRAM_API_KEY=
ASSEMBLYAI_API_KEY=
GOOGLE_APPLICATION_CREDENTIALS=

# LLM Providers
ANTHROPIC_API_KEY=

# Service Config
VOICE_ENGINE_PORT=8002
VOICE_ENGINE_HOST=0.0.0.0
```

### Basic Usage

```python
from voice_engine.tts import TTSProvider, ElevenLabsProvider
from voice_engine.stt import STTProvider, DeepgramProvider
from voice_engine.viseme import VisemeGenerator

# Text-to-Speech
tts = ElevenLabsProvider(voice_id="example_voice")
audio = await tts.synthesize("Hello, how can I help you?")

# Generate lip-sync data
viseme_gen = VisemeGenerator()
visemes = viseme_gen.generate(
    text="Hello, how can I help you?",
    audio=audio,
    fps=60
)

# Speech-to-Text
stt = DeepgramProvider()
async for transcript in stt.transcribe_stream(audio_stream):
    print(transcript.text)
```

## Viseme Generation

The engine generates MPEG-4 standard visemes (19 total):

| Viseme | Phonemes  | Description            |
| ------ | --------- | ---------------------- |
| sil    | -         | Silence/neutral        |
| PP     | p, b, m   | Bilabial plosive       |
| FF     | f, v      | Labiodental            |
| TH     | θ, ð      | Dental fricative       |
| DD     | t, d      | Alveolar plosive       |
| kk     | k, g      | Velar plosive          |
| CH     | tʃ, dʒ, ʃ | Palato-alveolar        |
| SS     | s, z      | Alveolar fricative     |
| nn     | n, l      | Alveolar nasal/lateral |
| RR     | r         | Alveolar approximant   |
| aa     | ɑ, æ      | Open vowels            |
| E      | e, ɛ      | Mid-front vowels       |
| ih     | ɪ, i      | Close-front vowels     |
| oh     | ɔ, o      | Mid-back vowels        |
| ou     | u, ʊ      | Close-back vowels      |

### Coarticulation Models

- **Direct**: No blending between visemes
- **Overlap**: Simple overlap blending
- **Dominance**: Vowel/consonant priority
- **Anticipatory**: Look-ahead blending

## TTS Providers

| Provider      | Streaming    | Voice Clone | Emotion |
| ------------- | ------------ | ----------- | ------- |
| ElevenLabs    | ✅ WebSocket | ✅          | ✅      |
| OpenAI TTS    | ❌ REST      | ❌          | ✅      |
| Cartesia      | ✅           | ✅          | ✅      |
| Deepgram Aura | ✅           | ❌          | ✅      |
| F5-TTS        | ✅           | ✅          | ✅      |
| Piper         | ✅ Local     | ❌          | ❌      |
| Orpheus       | ✅           | ✅          | ✅      |

## STT Providers

| Provider     | Streaming | Word Timing | Languages |
| ------------ | --------- | ----------- | --------- |
| Deepgram     | ✅        | ✅          | 30+       |
| Whisper      | ❌ Batch  | ✅          | 100+      |
| AssemblyAI   | ✅        | ✅          | 20+       |
| Google Cloud | ✅        | ✅          | 125+      |

## Audio Processing

### Voice Activity Detection (VAD)

- **WebRTC VAD**: Fast CPU-only (10-30ms frames)
- **Silero VAD**: Neural, high accuracy (30-60ms frames)
- **Energy-based**: Simple fallback

### Quality Features

- Echo cancellation (WebRTC AEC)
- Noise suppression
- Automatic gain control (AGC)
- Content classification

## Synchronization

The engine tracks A/V sync quality:

| Quality    | Offset    | Jitter  |
| ---------- | --------- | ------- |
| Excellent  | <20ms     | <5ms    |
| Good       | 20-50ms   | 5-15ms  |
| Acceptable | 50-100ms  | 15-30ms |
| Poor       | 100-200ms | >30ms   |

## Development

### Running the Server

```bash
# Development mode with auto-reload
nx serve psyche-voice-engine

# Production mode
nx serve-prod psyche-voice-engine
```

### Running Tests

```bash
# All tests
nx test psyche-voice-engine

# Skip provider tests (no API keys needed)
nx test-no-provider psyche-voice-engine

# With coverage
nx test-cov psyche-voice-engine
```

### Linting & Formatting

```bash
nx lint psyche-voice-engine
nx format psyche-voice-engine
```

### Docker

```bash
# Build GPU-enabled image
nx docker-build psyche-voice-engine

# Run with GPU support
nx docker-run psyche-voice-engine
```

## Nx Integration

This service is integrated with the Oshun Nx monorepo:

```bash
# Available targets
nx serve psyche-voice-engine           # Development server
nx serve-prod psyche-voice-engine      # Production server
nx build psyche-voice-engine           # Build package
nx install psyche-voice-engine         # Install dependencies
nx install-all-providers psyche-voice-engine # Install all SDKs
nx lint psyche-voice-engine            # Run linters
nx format psyche-voice-engine          # Format code
nx test psyche-voice-engine            # Run tests
nx test-no-provider psyche-voice-engine # Tests without providers
nx test-cov psyche-voice-engine        # Tests with coverage
nx docker-build psyche-voice-engine    # Build Docker image
nx docker-run psyche-voice-engine      # Run Docker container
```

## License

Proprietary - Oshun Platform