Real-time voice processing engine for TTS, STT, and conversational AI.
Part of the Psyche AI Virtual Assistant Platform.
Overview#
The Voice Engine provides comprehensive voice processing capabilities:
- Text-to-Speech (TTS) - 8 provider integrations with streaming support
- Speech-to-Text (STT) - 5 provider integrations with real-time transcription
- Viseme Generation - MPEG-4 compliant lip-sync with coarticulation
- Conversation Management - LLM integration and context handling
- Audio Processing - VAD, noise suppression, echo cancellation
Features#
- Multi-Provider TTS: ElevenLabs, OpenAI, Cartesia, Deepgram Aura, F5-TTS, Piper, Orpheus
- Multi-Provider STT: Deepgram, Whisper, AssemblyAI, Google Cloud Speech
- Real-Time Streaming: WebSocket-based audio streaming
- Lip-Sync Generation: 19-viseme MPEG-4 standard with blend shapes
- Natural Dialogue: Turn-taking, endpointing, interruption handling
- Failover Resilience: Circuit breaker pattern with health monitoring
- GPU Acceleration: CUDA-optimized for PyTorch models
Architecture#
text
voice_engine/
├── audio/ # Audio input processing
│ ├── capture.py # Audio capture
│ ├── vad.py # Voice activity detection
│ └── quality.py # Noise suppression, AGC
├── tts/ # Text-to-speech
│ ├── providers/ # Provider implementations
│ ├── streaming.py # Real-time streaming
│ └── cache.py # Response caching
├── stt/ # Speech-to-text
│ ├── providers/ # Provider implementations
│ └── streaming.py # Real-time transcription
├── viseme/ # Lip-sync generation
│ ├── generator.py # Viseme pipeline
│ └── coarticulation.py
├── phoneme/ # Phoneme processing
│ └── g2p.py # Grapheme-to-phoneme
├── conversation/ # Dialogue management
│ ├── llm.py # LLM integration
│ └── context.py # Context management
├── turn_taking/ # Natural dialogue flow
│ ├── endpointing.py # Utterance detection
│ └── interruption.py # Interruption handling
├── sync/ # A/V synchronization
│ └── timing.py # Drift correction
├── failover/ # Provider reliability
│ └── circuit_breaker.py
└── transcript/ # Transcript processing
├── streaming.py # Real-time updates
└── enhancement.py # Punctuation, entities
Quick Start#
Installation#
bash
# Using Nx
nx install psyche-voice-engine
# With all provider SDKs
nx install-all-providers psyche-voice-engine
# Or directly with Poetry
cd apps/psyche/voice-engine
poetry install
poetry install --extras all-providers
Environment Variables#
bash
# TTS Providers
ELEVENLABS_API_KEY=
OPENAI_API_KEY=
CARTESIA_API_KEY=
# STT Providers
DEEPGRAM_API_KEY=
ASSEMBLYAI_API_KEY=
GOOGLE_APPLICATION_CREDENTIALS=
# LLM Providers
ANTHROPIC_API_KEY=
# Service Config
VOICE_ENGINE_PORT=8002
VOICE_ENGINE_HOST=0.0.0.0
Basic Usage#
python
from voice_engine.tts import TTSProvider, ElevenLabsProvider
from voice_engine.stt import STTProvider, DeepgramProvider
from voice_engine.viseme import VisemeGenerator
# Text-to-Speech
tts = ElevenLabsProvider(voice_id="example_voice")
audio = await tts.synthesize("Hello, how can I help you?")
# Generate lip-sync data
viseme_gen = VisemeGenerator()
visemes = viseme_gen.generate(
text="Hello, how can I help you?",
audio=audio,
fps=60
)
# Speech-to-Text
stt = DeepgramProvider()
async for transcript in stt.transcribe_stream(audio_stream):
print(transcript.text)
Viseme Generation#
The engine generates MPEG-4 standard visemes (19 total):
| Viseme | Phonemes | Description |
|---|---|---|
| sil | - | Silence/neutral |
| PP | p, b, m | Bilabial plosive |
| FF | f, v | Labiodental |
| TH | θ, ð | Dental fricative |
| DD | t, d | Alveolar plosive |
| kk | k, g | Velar plosive |
| CH | tʃ, dʒ, ʃ | Palato-alveolar |
| SS | s, z | Alveolar fricative |
| nn | n, l | Alveolar nasal/lateral |
| RR | r | Alveolar approximant |
| aa | ɑ, æ | Open vowels |
| E | e, ɛ | Mid-front vowels |
| ih | ɪ, i | Close-front vowels |
| oh | ɔ, o | Mid-back vowels |
| ou | u, ʊ | Close-back vowels |
Coarticulation Models#
- Direct: No blending between visemes
- Overlap: Simple overlap blending
- Dominance: Vowel/consonant priority
- Anticipatory: Look-ahead blending
TTS Providers#
| Provider | Streaming | Voice Clone | Emotion |
|---|---|---|---|
| ElevenLabs | ✅ WebSocket | ✅ | ✅ |
| OpenAI TTS | ❌ REST | ❌ | ✅ |
| Cartesia | ✅ | ✅ | ✅ |
| Deepgram Aura | ✅ | ❌ | ✅ |
| F5-TTS | ✅ | ✅ | ✅ |
| Piper | ✅ Local | ❌ | ❌ |
| Orpheus | ✅ | ✅ | ✅ |
STT Providers#
| Provider | Streaming | Word Timing | Languages |
|---|---|---|---|
| Deepgram | ✅ | ✅ | 30+ |
| Whisper | ❌ Batch | ✅ | 100+ |
| AssemblyAI | ✅ | ✅ | 20+ |
| Google Cloud | ✅ | ✅ | 125+ |
Audio Processing#
Voice Activity Detection (VAD)#
- WebRTC VAD: Fast CPU-only (10-30ms frames)
- Silero VAD: Neural, high accuracy (30-60ms frames)
- Energy-based: Simple fallback
Quality Features#
- Echo cancellation (WebRTC AEC)
- Noise suppression
- Automatic gain control (AGC)
- Content classification
Synchronization#
The engine tracks A/V sync quality:
| Quality | Offset | Jitter |
|---|---|---|
| Excellent | <20ms | <5ms |
| Good | 20-50ms | 5-15ms |
| Acceptable | 50-100ms | 15-30ms |
| Poor | 100-200ms | >30ms |
Development#
Running the Server#
bash
# Development mode with auto-reload
nx serve psyche-voice-engine
# Production mode
nx serve-prod psyche-voice-engine
Running Tests#
bash
# All tests
nx test psyche-voice-engine
# Skip provider tests (no API keys needed)
nx test-no-provider psyche-voice-engine
# With coverage
nx test-cov psyche-voice-engine
Linting & Formatting#
bash
nx lint psyche-voice-engine
nx format psyche-voice-engine
Docker#
bash
# Build GPU-enabled image
nx docker-build psyche-voice-engine
# Run with GPU support
nx docker-run psyche-voice-engine
Nx Integration#
This service is integrated with the Oshun Nx monorepo:
bash
# Available targets
nx serve psyche-voice-engine # Development server
nx serve-prod psyche-voice-engine # Production server
nx build psyche-voice-engine # Build package
nx install psyche-voice-engine # Install dependencies
nx install-all-providers psyche-voice-engine # Install all SDKs
nx lint psyche-voice-engine # Run linters
nx format psyche-voice-engine # Format code
nx test psyche-voice-engine # Run tests
nx test-no-provider psyche-voice-engine # Tests without providers
nx test-cov psyche-voice-engine # Tests with coverage
nx docker-build psyche-voice-engine # Build Docker image
nx docker-run psyche-voice-engine # Run Docker container
License#
Proprietary - Oshun Platform