Overview#
The Lilith mobile app implements a complete real-time voice chat system with streaming Speech-to-Text (STT), bidirectional conversation with AI personas, streaming Text-to-Speech (TTS), and intelligent barge-in capabilities.
Architecture Diagram#
┌──────────────┐ WebSocket ┌──────────────┐
│ Mobile │◄─────────────────────────►│ Voice │
│ Client │ │ Pipeline │
│ │ HTTPS/REST │ Service │
│ │◄─────────────────────────►│ │
└──────────────┘ └──────────────┘
│ │
│ Audio Capture │
│ Audio Playback │
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ React │ │ Backend │
│ Native │ │ Services │
│ Audio API │ │ │
└──────────────┘ │ - STT │
│ - LLM │
│ - TTS │
└──────────────┘
Core Components#
1. WebSocketClient (src/services/WebSocketClient.ts)#
Purpose: Provides reliable WebSocket connectivity with automatic reconnection, message queuing, and stream management.
Key Features:
- Automatic Reconnection: Exponential backoff with stability tracking
- Message Queuing: Queues messages when disconnected, sends when reconnected
- Stream Abstraction: Creates bidirectional streams for STT/TTS
- Request/Response Pattern: Async communication with timeout handling
- Ping/Pong Keep-Alive: Heartbeat mechanism to maintain connections
Configuration:
const wsConfig = {
baseUrl: 'wss://api.lilith.app/ws',
reconnect: true,
reconnectInterval: 1000, // Base delay in ms
reconnectAttempts: 5, // Max reconnection attempts
timeout: 30000, // Request timeout in ms
pingInterval: 30000, // Heartbeat interval in ms
};
API:
// Connect to WebSocket server
await webSocketClient.connect();
// Send a message
webSocketClient.send({
type: 'message',
data: { content: 'Hello' }
});
// Request/response pattern
const response = await webSocketClient.request({
type: 'query',
data: { query: 'status' }
});
// Create a streaming connection
const stream = webSocketClient.createStream({
endpoint: '/v1/stt/stream',
params: { language: 'en-US' }
});
// Handle stream messages
stream.onmessage = (event) => {
console.log('Received:', event.data);
};
// Send data to stream
stream.send({ audio: audioData });
// Close stream
stream.close();
// Disconnect
await webSocketClient.disconnect();
Events:
connected: WebSocket connection establisheddisconnected: WebSocket connection closedconnecting: Attempting to connecterror: Error occurredreconnecting: Attempting to reconnect (includes attempt number, max attempts, delay)reconnectFailed: Max reconnection attempts reachedmessage: General message receivedstreamMessage: Stream-specific message receivedstreamError: Stream error occurredpong: Pong response to ping receivedmessageSent: Message successfully sent
Reconnection Logic:
// Connection stability tracking prevents infinite reconnect loops
// Only resets attempt counter after connection proves stable (1 second)
private scheduleReconnect(): void {
this.isInReconnectCycle = true;
this.reconnectAttempt++;
const delay = this.config.reconnectInterval * Math.pow(2, this.reconnectAttempt - 1);
setTimeout(() => this.connect(), delay);
}
Test Coverage: 27/27 tests passing (100%)
- Connection management
- Message handling
- Request/response pattern
- Streaming interface
- Reconnection with exponential backoff
- Ping/pong keep-alive
- Error handling
- Resource cleanup
2. VoicePipelineService (src/services/VoicePipelineService.ts)#
Purpose: Orchestrates end-to-end voice conversations (STT → LLM → TTS) with session management and barge-in support.
Key Features:
- Session Management: Create and manage voice conversation sessions
- STT Streaming: Real-time speech-to-text transcription with VAD
- TTS Streaming: Chunked audio synthesis with progress tracking
- Barge-In: Interrupt AI speech when user starts speaking
- Connection Lifecycle: Automatic initialization and cleanup
Session Types:
type SessionMode = 'push-to-talk' | 'vad' | 'hybrid';
interface VoiceSession {
id: string;
userId: string;
personaId?: string;
mode: SessionMode;
vadConfig?: {
threshold: number; // 0.0-1.0, default 0.7
silenceTimeout: number; // ms, default 1500
};
bargeInConfig?: {
enabled: boolean;
threshold: number; // 0.0-1.0, default 0.8
fadeOutDuration: number; // ms, default 200
};
status: 'active' | 'paused' | 'completed';
}
API:
// Initialize service
await voicePipelineService.initialize();
// Create a voice session
const session = await voicePipelineService.createSession({
userId: 'user-123',
personaId: 'zen-master',
mode: 'hybrid',
vadConfig: {
threshold: 0.7,
silenceTimeout: 1500
},
bargeInConfig: {
enabled: true,
threshold: 0.8,
fadeOutDuration: 200
}
});
// Start STT streaming
const sttStream = voicePipelineService.startSTTStream({
language: 'en-US',
model: 'whisper-small',
enablePartialResults: true,
enableVAD: true
});
// Send audio to STT
sttStream.sendAudio(audioBufferData);
// Listen for transcriptions
voicePipelineService.on('transcription:partial', (result) => {
console.log('Partial:', result.text);
});
voicePipelineService.on('transcription:final', (result) => {
console.log('Final:', result.text, 'Confidence:', result.confidence);
});
// Stop STT
sttStream.stop();
// Start TTS streaming
const ttsStream = voicePipelineService.startTTSStream(
'Hello, how can I help you today?',
{
voiceId: 'voice-zen-master',
language: 'en-US',
speed: 1.0,
pitch: 1.0,
format: 'opus'
}
);
// Handle TTS chunks
ttsStream.onChunk((chunk) => {
playAudio(chunk.audio);
console.log('Progress:', chunk.progress);
});
// Handle completion
ttsStream.onComplete(() => {
console.log('TTS complete');
});
// Trigger barge-in
await voicePipelineService.triggerBargeIn(audioLevel);
// End session
await voicePipelineService.endSession();
// Cleanup
await voicePipelineService.dispose();
Events:
initialized: Service initializedconnected: WebSocket connecteddisconnected: WebSocket disconnectedsessionCreated: Voice session createdsessionEnded: Voice session endedtranscription:partial: Partial transcription resulttranscription:final: Final transcription resultvad:activity: Voice activity detected (active, audioLevel, timestamp)stt:error: STT streaming errorstt:stopped: STT streaming stoppedtts:chunk: TTS audio chunk receivedtts:complete: TTS streaming completetts:error: TTS streaming errortts:cancelled: TTS streaming cancelledbargeIn:triggered: Barge-in event triggered (interrupted, timestamp, audioLevel)error: General errorreconnecting: WebSocket reconnectingreconnectFailed: WebSocket reconnection failed
Barge-In Logic:
// When barge-in is triggered:
// 1. API call to backend to validate and log barge-in event
// 2. If successful, stops all active TTS streams
// 3. Emits bargeIn:triggered event with details
async triggerBargeIn(audioLevel: number): Promise<void> {
const response = await fetch(
`${this.config.apiBaseUrl}/v1/voice/sessions/${session.id}/barge-in`,
{
method: 'POST',
body: JSON.stringify({ audioLevel, timestamp: Date.now() })
}
);
const result: BargeInEvent = await response.json();
if (result.interrupted) {
// Stop all active TTS streams
this.activeStreams.forEach((stream, streamId) => {
if (streamId.startsWith('tts-')) {
stream.close();
this.activeStreams.delete(streamId);
}
});
}
}
Test Coverage: 22/22 tests passing (100%)
- Service initialization
- Session creation and management
- STT streaming (transcription, VAD)
- TTS streaming (chunks, completion)
- Barge-in functionality
- Connection status
- Error handling
- Resource cleanup
3. Service Integration (src/services/serviceInstances.ts)#
Purpose: Centralized service initialization with proper configuration and dependency injection.
Configuration:
const API_BASE_URL = process.env.REACT_APP_API_BASE_URL || 'https://api.lilith.app';
const WS_BASE_URL = process.env.REACT_APP_WS_BASE_URL || 'wss://api.lilith.app/ws';
export const webSocketClient = new WebSocketClient({
baseUrl: WS_BASE_URL,
reconnect: true,
reconnectInterval: 1000,
reconnectAttempts: 5,
timeout: 30000,
pingInterval: 30000,
});
export const voicePipelineService = new VoicePipelineService({
apiBaseUrl: API_BASE_URL,
wsBaseUrl: WS_BASE_URL,
sttEndpoint: '/v1/stt/transcribe',
ttsEndpoint: '/v1/tts/synthesize',
voiceSessionEndpoint: '/v1/voice/sessions',
webSocketConfig: {
reconnect: true,
reconnectInterval: 1000,
reconnectAttempts: 5,
timeout: 30000,
pingInterval: 30000,
},
});
Usage Examples#
Basic Voice Chat Flow#
import { voicePipelineService } from '../services/serviceInstances';
import { AudioRecorder } from 'react-native-audio-recorder-player';
const audioRecorder = new AudioRecorder();
// 1. Initialize and create session
await voicePipelineService.initialize();
const session = await voicePipelineService.createSession({
userId: 'user-123',
personaId: 'zen-master',
mode: 'hybrid'
});
// 2. Start recording and STT streaming
const sttStream = voicePipelineService.startSTTStream();
audioRecorder.startRecording();
audioRecorder.addRecordBackListener((e) => {
// Send audio chunks to STT
sttStream.sendAudio(e.currentMetering);
});
// 3. Listen for transcription
voicePipelineService.on('transcription:final', async (result) => {
console.log('User said:', result.text);
// Stop recording
await audioRecorder.stopRecording();
sttStream.stop();
// Get AI response (via separate LLM API call)
const aiResponse = await fetch(`${API_BASE_URL}/v1/chat`, {
method: 'POST',
body: JSON.stringify({
message: result.text,
personaId: 'zen-master'
})
}).then(r => r.json());
// 4. Start TTS playback
const ttsStream = voicePipelineService.startTTSStream(aiResponse.text);
const audioPlayer = new AudioRecorder();
const audioChunks: ArrayBuffer[] = [];
ttsStream.onChunk((chunk) => {
audioChunks.push(chunk.audio);
});
ttsStream.onComplete(() => {
// Combine chunks and play
const fullAudio = combineAudioChunks(audioChunks);
audioPlayer.startPlayer(fullAudio);
});
});
// 5. Handle barge-in
audioRecorder.addRecordBackListener((e) => {
if (e.currentMetering > -30) { // User is speaking
voicePipelineService.triggerBargeIn(e.currentMetering);
// Restart recording flow
}
});
// 6. Cleanup
await voicePipelineService.endSession();
await voicePipelineService.dispose();
Push-to-Talk Mode#
// Use push-to-talk mode for simpler interaction
const session = await voicePipelineService.createSession({
userId: 'user-123',
mode: 'push-to-talk' // No VAD, user controls recording
});
// User presses button
function onPressStart() {
const sttStream = voicePipelineService.startSTTStream({
enableVAD: false // Disable VAD in push-to-talk
});
audioRecorder.startRecording();
// ... send audio to sttStream
}
// User releases button
function onPressEnd() {
audioRecorder.stopRecording();
sttStream.stop();
}
VAD (Voice Activity Detection) Mode#
// Use VAD mode for hands-free operation
const session = await voicePipelineService.createSession({
userId: 'user-123',
mode: 'vad',
vadConfig: {
threshold: 0.7, // Sensitivity (0.0 = any sound, 1.0 = only loud sounds)
silenceTimeout: 1500 // ms of silence before ending transcription
}
});
// Listen for VAD events
voicePipelineService.on('vad:activity', (event) => {
if (event.active) {
console.log('User started speaking');
// Show speaking indicator
} else {
console.log('User stopped speaking');
// Hide speaking indicator
}
});
Error Handling#
Connection Errors#
voicePipelineService.on('error', (error) => {
console.error('Service error:', error);
// Show error UI
});
voicePipelineService.on('reconnecting', (info) => {
console.log(`Reconnecting attempt ${info.attempt}/${info.maxAttempts}`);
// Show reconnecting indicator
});
voicePipelineService.on('reconnectFailed', () => {
console.error('Failed to reconnect');
// Show offline mode or retry button
});
Stream Errors#
// STT errors
voicePipelineService.on('stt:error', (error) => {
console.error('STT error:', error);
// Restart STT stream or show error
});
// TTS errors
const ttsStream = voicePipelineService.startTTSStream(text);
ttsStream.onError((error) => {
console.error('TTS error:', error);
// Retry or show error
});
Performance Considerations#
Audio Chunk Size#
// Optimize chunk size for network conditions
const STT_CHUNK_SIZE = 4096; // bytes
const TTS_CHUNK_SIZE = 4096; // bytes
// Smaller chunks = more real-time, more overhead
// Larger chunks = less real-time, less overhead
Memory Management#
// Clear audio buffers when done
audioChunks = [];
// Dispose services when component unmounts
useEffect(() => {
return () => {
voicePipelineService.dispose();
};
}, []);
Network Optimization#
// Use opus format for smaller payload size
const ttsStream = voicePipelineService.startTTSStream(text, {
format: 'opus' // ~50% smaller than wav
});
// Enable compression in STT
const sttStream = voicePipelineService.startSTTStream({
enablePartialResults: true // Get results faster
});
Testing#
Unit Tests#
# Run WebSocketClient tests
npm test -- src/__tests__/services/WebSocketClient.test.ts
# Run VoicePipelineService tests
npm test -- src/__tests__/services/VoicePipelineService.test.ts
Integration Tests#
# Run all voice service tests
npm test -- src/__tests__/services/
Manual Testing#
- Connection Test: Verify WebSocket connects successfully
- STT Test: Speak into microphone, verify transcription appears
- TTS Test: Send text, verify audio plays correctly
- Barge-In Test: Speak while AI is talking, verify AI stops
- Reconnection Test: Disable network, re-enable, verify reconnection
- Error Test: Send invalid data, verify error handling
Security Considerations#
Authentication#
// Include auth token in WebSocket connection
const webSocketClient = new WebSocketClient({
baseUrl: `${WS_BASE_URL}?token=${authToken}`,
// ...
});
// Include auth token in API calls
const session = await voicePipelineService.createSession({
userId: 'user-123',
// Session will use authenticated user from token
});
Data Privacy#
- Audio is streamed, not stored on client
- Transcriptions are ephemeral (not persisted unless explicitly saved)
- Session data includes minimal PII
- All connections use WSS (WebSocket Secure) and HTTPS
Rate Limiting#
// Handle rate limit errors
voicePipelineService.on('error', (error) => {
if (error.code === 'RATE_LIMIT_EXCEEDED') {
// Show rate limit message
// Wait before retrying
}
});
Troubleshooting#
Common Issues#
Issue: WebSocket connection fails
Solution: Check WS_BASE_URL configuration, verify network connectivity, check authentication token
Issue: Transcription is slow or inaccurate Solution: Check audio quality, adjust VAD threshold, use appropriate STT model size
Issue: TTS audio is choppy Solution: Increase chunk size, check network speed, enable audio buffering
Issue: Barge-in not working
Solution: Verify bargeInConfig.enabled = true, check audio level threshold, ensure STT stream is active
Issue: Memory leak in long sessions
Solution: Call dispose() when done, clear audio buffers regularly, use endSession() between conversations
Debug Logging#
// Enable debug logging
voicePipelineService.on('transcription:partial', (result) => {
console.log('[STT Partial]', result.text, result.confidence);
});
voicePipelineService.on('tts:chunk', (chunk) => {
console.log('[TTS Chunk]', chunk.chunkIndex, '/', chunk.totalChunks);
});
webSocketClient.on('messageSent', (message) => {
console.log('[WS Sent]', message.type);
});
Future Enhancements#
- Multi-language support with automatic language detection
- Emotion detection from voice tone
- Speaker diarization for multi-user conversations
- Audio effects (reverb, EQ) for persona voice customization
- Offline mode with local STT/TTS models
- Advanced VAD with noise cancellation
- Real-time translation between languages