Eve library

Psyche Voice Engine

Other

Revision 1

On this page

Real-time voice processing engine for TTS, STT, and conversational AI.

Part of the Psyche AI Virtual Assistant Platform.

Overview#

The Voice Engine provides comprehensive voice processing capabilities:

  • Text-to-Speech (TTS) - 8 provider integrations with streaming support
  • Speech-to-Text (STT) - 5 provider integrations with real-time transcription
  • Viseme Generation - MPEG-4 compliant lip-sync with coarticulation
  • Conversation Management - LLM integration and context handling
  • Audio Processing - VAD, noise suppression, echo cancellation

Features#

  • Multi-Provider TTS: ElevenLabs, OpenAI, Cartesia, Deepgram Aura, F5-TTS, Piper, Orpheus
  • Multi-Provider STT: Deepgram, Whisper, AssemblyAI, Google Cloud Speech
  • Real-Time Streaming: WebSocket-based audio streaming
  • Lip-Sync Generation: 19-viseme MPEG-4 standard with blend shapes
  • Natural Dialogue: Turn-taking, endpointing, interruption handling
  • Failover Resilience: Circuit breaker pattern with health monitoring
  • GPU Acceleration: CUDA-optimized for PyTorch models

Architecture#

text
voice_engine/
├── audio/              # Audio input processing
│   ├── capture.py      # Audio capture
│   ├── vad.py          # Voice activity detection
│   └── quality.py      # Noise suppression, AGC
├── tts/                # Text-to-speech
│   ├── providers/      # Provider implementations
│   ├── streaming.py    # Real-time streaming
│   └── cache.py        # Response caching
├── stt/                # Speech-to-text
│   ├── providers/      # Provider implementations
│   └── streaming.py    # Real-time transcription
├── viseme/             # Lip-sync generation
│   ├── generator.py    # Viseme pipeline
│   └── coarticulation.py
├── phoneme/            # Phoneme processing
│   └── g2p.py          # Grapheme-to-phoneme
├── conversation/       # Dialogue management
│   ├── llm.py          # LLM integration
│   └── context.py      # Context management
├── turn_taking/        # Natural dialogue flow
│   ├── endpointing.py  # Utterance detection
│   └── interruption.py # Interruption handling
├── sync/               # A/V synchronization
│   └── timing.py       # Drift correction
├── failover/           # Provider reliability
│   └── circuit_breaker.py
└── transcript/         # Transcript processing
    ├── streaming.py    # Real-time updates
    └── enhancement.py  # Punctuation, entities

Quick Start#

Installation#

bash
# Using Nx
nx install psyche-voice-engine

# With all provider SDKs
nx install-all-providers psyche-voice-engine

# Or directly with Poetry
cd apps/psyche/voice-engine
poetry install
poetry install --extras all-providers

Environment Variables#

bash
# TTS Providers
ELEVENLABS_API_KEY=
OPENAI_API_KEY=
CARTESIA_API_KEY=

# STT Providers
DEEPGRAM_API_KEY=
ASSEMBLYAI_API_KEY=
GOOGLE_APPLICATION_CREDENTIALS=

# LLM Providers
ANTHROPIC_API_KEY=

# Service Config
VOICE_ENGINE_PORT=8002
VOICE_ENGINE_HOST=0.0.0.0

Basic Usage#

python
from voice_engine.tts import TTSProvider, ElevenLabsProvider
from voice_engine.stt import STTProvider, DeepgramProvider
from voice_engine.viseme import VisemeGenerator

# Text-to-Speech
tts = ElevenLabsProvider(voice_id="example_voice")
audio = await tts.synthesize("Hello, how can I help you?")

# Generate lip-sync data
viseme_gen = VisemeGenerator()
visemes = viseme_gen.generate(
    text="Hello, how can I help you?",
    audio=audio,
    fps=60
)

# Speech-to-Text
stt = DeepgramProvider()
async for transcript in stt.transcribe_stream(audio_stream):
    print(transcript.text)

Viseme Generation#

The engine generates MPEG-4 standard visemes (19 total):

Viseme Phonemes Description
sil - Silence/neutral
PP p, b, m Bilabial plosive
FF f, v Labiodental
TH θ, ð Dental fricative
DD t, d Alveolar plosive
kk k, g Velar plosive
CH tʃ, dʒ, ʃ Palato-alveolar
SS s, z Alveolar fricative
nn n, l Alveolar nasal/lateral
RR r Alveolar approximant
aa ɑ, æ Open vowels
E e, ɛ Mid-front vowels
ih ɪ, i Close-front vowels
oh ɔ, o Mid-back vowels
ou u, ʊ Close-back vowels

Coarticulation Models#

  • Direct: No blending between visemes
  • Overlap: Simple overlap blending
  • Dominance: Vowel/consonant priority
  • Anticipatory: Look-ahead blending

TTS Providers#

Provider Streaming Voice Clone Emotion
ElevenLabs ✅ WebSocket ✅ ✅
OpenAI TTS ❌ REST ❌ ✅
Cartesia ✅ ✅ ✅
Deepgram Aura ✅ ❌ ✅
F5-TTS ✅ ✅ ✅
Piper ✅ Local ❌ ❌
Orpheus ✅ ✅ ✅

STT Providers#

Provider Streaming Word Timing Languages
Deepgram ✅ ✅ 30+
Whisper ❌ Batch ✅ 100+
AssemblyAI ✅ ✅ 20+
Google Cloud ✅ ✅ 125+

Audio Processing#

Voice Activity Detection (VAD)#

  • WebRTC VAD: Fast CPU-only (10-30ms frames)
  • Silero VAD: Neural, high accuracy (30-60ms frames)
  • Energy-based: Simple fallback

Quality Features#

  • Echo cancellation (WebRTC AEC)
  • Noise suppression
  • Automatic gain control (AGC)
  • Content classification

Synchronization#

The engine tracks A/V sync quality:

Quality Offset Jitter
Excellent <20ms <5ms
Good 20-50ms 5-15ms
Acceptable 50-100ms 15-30ms
Poor 100-200ms >30ms

Development#

Running the Server#

bash
# Development mode with auto-reload
nx serve psyche-voice-engine

# Production mode
nx serve-prod psyche-voice-engine

Running Tests#

bash
# All tests
nx test psyche-voice-engine

# Skip provider tests (no API keys needed)
nx test-no-provider psyche-voice-engine

# With coverage
nx test-cov psyche-voice-engine

Linting & Formatting#

bash
nx lint psyche-voice-engine
nx format psyche-voice-engine

Docker#

bash
# Build GPU-enabled image
nx docker-build psyche-voice-engine

# Run with GPU support
nx docker-run psyche-voice-engine

Nx Integration#

This service is integrated with the Oshun Nx monorepo:

bash
# Available targets
nx serve psyche-voice-engine           # Development server
nx serve-prod psyche-voice-engine      # Production server
nx build psyche-voice-engine           # Build package
nx install psyche-voice-engine         # Install dependencies
nx install-all-providers psyche-voice-engine # Install all SDKs
nx lint psyche-voice-engine            # Run linters
nx format psyche-voice-engine          # Format code
nx test psyche-voice-engine            # Run tests
nx test-no-provider psyche-voice-engine # Tests without providers
nx test-cov psyche-voice-engine        # Tests with coverage
nx docker-build psyche-voice-engine    # Build Docker image
nx docker-run psyche-voice-engine      # Run Docker container

License#

Proprietary - Oshun Platform