Eve library

Voice Chat Architecture

Guide

Revision 1

On this page

Overview#

The Lilith mobile app implements a complete real-time voice chat system with streaming Speech-to-Text (STT), bidirectional conversation with AI personas, streaming Text-to-Speech (TTS), and intelligent barge-in capabilities.

Architecture Diagram#

text
┌──────────────┐        WebSocket         ┌──────────────┐
│   Mobile     │◄─────────────────────────►│  Voice       │
│   Client     │                           │  Pipeline    │
│              │        HTTPS/REST         │  Service     │
│              │◄─────────────────────────►│              │
└──────────────┘                           └──────────────┘
       │                                           │
       │ Audio Capture                             │
       │ Audio Playback                            │
       │                                           │
       ▼                                           ▼
┌──────────────┐                           ┌──────────────┐
│  React       │                           │   Backend    │
│  Native      │                           │   Services   │
│  Audio API   │                           │              │
└──────────────┘                           │  - STT       │
                                            │  - LLM       │
                                            │  - TTS       │
                                            └──────────────┘

Core Components#

1. WebSocketClient (src/services/WebSocketClient.ts)#

Purpose: Provides reliable WebSocket connectivity with automatic reconnection, message queuing, and stream management.

Key Features:

  • Automatic Reconnection: Exponential backoff with stability tracking
  • Message Queuing: Queues messages when disconnected, sends when reconnected
  • Stream Abstraction: Creates bidirectional streams for STT/TTS
  • Request/Response Pattern: Async communication with timeout handling
  • Ping/Pong Keep-Alive: Heartbeat mechanism to maintain connections

Configuration:

typescript
const wsConfig = {
  baseUrl: 'wss://api.lilith.app/ws',
  reconnect: true,
  reconnectInterval: 1000,      // Base delay in ms
  reconnectAttempts: 5,          // Max reconnection attempts
  timeout: 30000,                // Request timeout in ms
  pingInterval: 30000,           // Heartbeat interval in ms
};

API:

typescript
// Connect to WebSocket server
await webSocketClient.connect();

// Send a message
webSocketClient.send({
  type: 'message',
  data: { content: 'Hello' }
});

// Request/response pattern
const response = await webSocketClient.request({
  type: 'query',
  data: { query: 'status' }
});

// Create a streaming connection
const stream = webSocketClient.createStream({
  endpoint: '/v1/stt/stream',
  params: { language: 'en-US' }
});

// Handle stream messages
stream.onmessage = (event) => {
  console.log('Received:', event.data);
};

// Send data to stream
stream.send({ audio: audioData });

// Close stream
stream.close();

// Disconnect
await webSocketClient.disconnect();

Events:

  • connected: WebSocket connection established
  • disconnected: WebSocket connection closed
  • connecting: Attempting to connect
  • error: Error occurred
  • reconnecting: Attempting to reconnect (includes attempt number, max attempts, delay)
  • reconnectFailed: Max reconnection attempts reached
  • message: General message received
  • streamMessage: Stream-specific message received
  • streamError: Stream error occurred
  • pong: Pong response to ping received
  • messageSent: Message successfully sent

Reconnection Logic:

typescript
// Connection stability tracking prevents infinite reconnect loops
// Only resets attempt counter after connection proves stable (1 second)
private scheduleReconnect(): void {
  this.isInReconnectCycle = true;
  this.reconnectAttempt++;
  const delay = this.config.reconnectInterval * Math.pow(2, this.reconnectAttempt - 1);

  setTimeout(() => this.connect(), delay);
}

Test Coverage: 27/27 tests passing (100%)

  • Connection management
  • Message handling
  • Request/response pattern
  • Streaming interface
  • Reconnection with exponential backoff
  • Ping/pong keep-alive
  • Error handling
  • Resource cleanup

2. VoicePipelineService (src/services/VoicePipelineService.ts)#

Purpose: Orchestrates end-to-end voice conversations (STT → LLM → TTS) with session management and barge-in support.

Key Features:

  • Session Management: Create and manage voice conversation sessions
  • STT Streaming: Real-time speech-to-text transcription with VAD
  • TTS Streaming: Chunked audio synthesis with progress tracking
  • Barge-In: Interrupt AI speech when user starts speaking
  • Connection Lifecycle: Automatic initialization and cleanup

Session Types:

typescript
type SessionMode = 'push-to-talk' | 'vad' | 'hybrid';

interface VoiceSession {
  id: string;
  userId: string;
  personaId?: string;
  mode: SessionMode;
  vadConfig?: {
    threshold: number;        // 0.0-1.0, default 0.7
    silenceTimeout: number;   // ms, default 1500
  };
  bargeInConfig?: {
    enabled: boolean;
    threshold: number;        // 0.0-1.0, default 0.8
    fadeOutDuration: number;  // ms, default 200
  };
  status: 'active' | 'paused' | 'completed';
}

API:

typescript
// Initialize service
await voicePipelineService.initialize();

// Create a voice session
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  personaId: 'zen-master',
  mode: 'hybrid',
  vadConfig: {
    threshold: 0.7,
    silenceTimeout: 1500
  },
  bargeInConfig: {
    enabled: true,
    threshold: 0.8,
    fadeOutDuration: 200
  }
});

// Start STT streaming
const sttStream = voicePipelineService.startSTTStream({
  language: 'en-US',
  model: 'whisper-small',
  enablePartialResults: true,
  enableVAD: true
});

// Send audio to STT
sttStream.sendAudio(audioBufferData);

// Listen for transcriptions
voicePipelineService.on('transcription:partial', (result) => {
  console.log('Partial:', result.text);
});

voicePipelineService.on('transcription:final', (result) => {
  console.log('Final:', result.text, 'Confidence:', result.confidence);
});

// Stop STT
sttStream.stop();

// Start TTS streaming
const ttsStream = voicePipelineService.startTTSStream(
  'Hello, how can I help you today?',
  {
    voiceId: 'voice-zen-master',
    language: 'en-US',
    speed: 1.0,
    pitch: 1.0,
    format: 'opus'
  }
);

// Handle TTS chunks
ttsStream.onChunk((chunk) => {
  playAudio(chunk.audio);
  console.log('Progress:', chunk.progress);
});

// Handle completion
ttsStream.onComplete(() => {
  console.log('TTS complete');
});

// Trigger barge-in
await voicePipelineService.triggerBargeIn(audioLevel);

// End session
await voicePipelineService.endSession();

// Cleanup
await voicePipelineService.dispose();

Events:

  • initialized: Service initialized
  • connected: WebSocket connected
  • disconnected: WebSocket disconnected
  • sessionCreated: Voice session created
  • sessionEnded: Voice session ended
  • transcription:partial: Partial transcription result
  • transcription:final: Final transcription result
  • vad:activity: Voice activity detected (active, audioLevel, timestamp)
  • stt:error: STT streaming error
  • stt:stopped: STT streaming stopped
  • tts:chunk: TTS audio chunk received
  • tts:complete: TTS streaming complete
  • tts:error: TTS streaming error
  • tts:cancelled: TTS streaming cancelled
  • bargeIn:triggered: Barge-in event triggered (interrupted, timestamp, audioLevel)
  • error: General error
  • reconnecting: WebSocket reconnecting
  • reconnectFailed: WebSocket reconnection failed

Barge-In Logic:

typescript
// When barge-in is triggered:
// 1. API call to backend to validate and log barge-in event
// 2. If successful, stops all active TTS streams
// 3. Emits bargeIn:triggered event with details

async triggerBargeIn(audioLevel: number): Promise<void> {
  const response = await fetch(
    `${this.config.apiBaseUrl}/v1/voice/sessions/${session.id}/barge-in`,
    {
      method: 'POST',
      body: JSON.stringify({ audioLevel, timestamp: Date.now() })
    }
  );

  const result: BargeInEvent = await response.json();

  if (result.interrupted) {
    // Stop all active TTS streams
    this.activeStreams.forEach((stream, streamId) => {
      if (streamId.startsWith('tts-')) {
        stream.close();
        this.activeStreams.delete(streamId);
      }
    });
  }
}

Test Coverage: 22/22 tests passing (100%)

  • Service initialization
  • Session creation and management
  • STT streaming (transcription, VAD)
  • TTS streaming (chunks, completion)
  • Barge-in functionality
  • Connection status
  • Error handling
  • Resource cleanup

3. Service Integration (src/services/serviceInstances.ts)#

Purpose: Centralized service initialization with proper configuration and dependency injection.

Configuration:

typescript
const API_BASE_URL = process.env.REACT_APP_API_BASE_URL || 'https://api.lilith.app';
const WS_BASE_URL = process.env.REACT_APP_WS_BASE_URL || 'wss://api.lilith.app/ws';

export const webSocketClient = new WebSocketClient({
  baseUrl: WS_BASE_URL,
  reconnect: true,
  reconnectInterval: 1000,
  reconnectAttempts: 5,
  timeout: 30000,
  pingInterval: 30000,
});

export const voicePipelineService = new VoicePipelineService({
  apiBaseUrl: API_BASE_URL,
  wsBaseUrl: WS_BASE_URL,
  sttEndpoint: '/v1/stt/transcribe',
  ttsEndpoint: '/v1/tts/synthesize',
  voiceSessionEndpoint: '/v1/voice/sessions',
  webSocketConfig: {
    reconnect: true,
    reconnectInterval: 1000,
    reconnectAttempts: 5,
    timeout: 30000,
    pingInterval: 30000,
  },
});

Usage Examples#

Basic Voice Chat Flow#

typescript
import { voicePipelineService } from '../services/serviceInstances';
import { AudioRecorder } from 'react-native-audio-recorder-player';

const audioRecorder = new AudioRecorder();

// 1. Initialize and create session
await voicePipelineService.initialize();
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  personaId: 'zen-master',
  mode: 'hybrid'
});

// 2. Start recording and STT streaming
const sttStream = voicePipelineService.startSTTStream();

audioRecorder.startRecording();
audioRecorder.addRecordBackListener((e) => {
  // Send audio chunks to STT
  sttStream.sendAudio(e.currentMetering);
});

// 3. Listen for transcription
voicePipelineService.on('transcription:final', async (result) => {
  console.log('User said:', result.text);

  // Stop recording
  await audioRecorder.stopRecording();
  sttStream.stop();

  // Get AI response (via separate LLM API call)
  const aiResponse = await fetch(`${API_BASE_URL}/v1/chat`, {
    method: 'POST',
    body: JSON.stringify({
      message: result.text,
      personaId: 'zen-master'
    })
  }).then(r => r.json());

  // 4. Start TTS playback
  const ttsStream = voicePipelineService.startTTSStream(aiResponse.text);

  const audioPlayer = new AudioRecorder();
  const audioChunks: ArrayBuffer[] = [];

  ttsStream.onChunk((chunk) => {
    audioChunks.push(chunk.audio);
  });

  ttsStream.onComplete(() => {
    // Combine chunks and play
    const fullAudio = combineAudioChunks(audioChunks);
    audioPlayer.startPlayer(fullAudio);
  });
});

// 5. Handle barge-in
audioRecorder.addRecordBackListener((e) => {
  if (e.currentMetering > -30) { // User is speaking
    voicePipelineService.triggerBargeIn(e.currentMetering);
    // Restart recording flow
  }
});

// 6. Cleanup
await voicePipelineService.endSession();
await voicePipelineService.dispose();

Push-to-Talk Mode#

typescript
// Use push-to-talk mode for simpler interaction
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  mode: 'push-to-talk'  // No VAD, user controls recording
});

// User presses button
function onPressStart() {
  const sttStream = voicePipelineService.startSTTStream({
    enableVAD: false  // Disable VAD in push-to-talk
  });

  audioRecorder.startRecording();
  // ... send audio to sttStream
}

// User releases button
function onPressEnd() {
  audioRecorder.stopRecording();
  sttStream.stop();
}

VAD (Voice Activity Detection) Mode#

typescript
// Use VAD mode for hands-free operation
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  mode: 'vad',
  vadConfig: {
    threshold: 0.7,      // Sensitivity (0.0 = any sound, 1.0 = only loud sounds)
    silenceTimeout: 1500  // ms of silence before ending transcription
  }
});

// Listen for VAD events
voicePipelineService.on('vad:activity', (event) => {
  if (event.active) {
    console.log('User started speaking');
    // Show speaking indicator
  } else {
    console.log('User stopped speaking');
    // Hide speaking indicator
  }
});

Error Handling#

Connection Errors#

typescript
voicePipelineService.on('error', (error) => {
  console.error('Service error:', error);
  // Show error UI
});

voicePipelineService.on('reconnecting', (info) => {
  console.log(`Reconnecting attempt ${info.attempt}/${info.maxAttempts}`);
  // Show reconnecting indicator
});

voicePipelineService.on('reconnectFailed', () => {
  console.error('Failed to reconnect');
  // Show offline mode or retry button
});

Stream Errors#

typescript
// STT errors
voicePipelineService.on('stt:error', (error) => {
  console.error('STT error:', error);
  // Restart STT stream or show error
});

// TTS errors
const ttsStream = voicePipelineService.startTTSStream(text);
ttsStream.onError((error) => {
  console.error('TTS error:', error);
  // Retry or show error
});

Performance Considerations#

Audio Chunk Size#

typescript
// Optimize chunk size for network conditions
const STT_CHUNK_SIZE = 4096;  // bytes
const TTS_CHUNK_SIZE = 4096;  // bytes

// Smaller chunks = more real-time, more overhead
// Larger chunks = less real-time, less overhead

Memory Management#

typescript
// Clear audio buffers when done
audioChunks = [];

// Dispose services when component unmounts
useEffect(() => {
  return () => {
    voicePipelineService.dispose();
  };
}, []);

Network Optimization#

typescript
// Use opus format for smaller payload size
const ttsStream = voicePipelineService.startTTSStream(text, {
  format: 'opus'  // ~50% smaller than wav
});

// Enable compression in STT
const sttStream = voicePipelineService.startSTTStream({
  enablePartialResults: true  // Get results faster
});

Testing#

Unit Tests#

bash
# Run WebSocketClient tests
npm test -- src/__tests__/services/WebSocketClient.test.ts

# Run VoicePipelineService tests
npm test -- src/__tests__/services/VoicePipelineService.test.ts

Integration Tests#

bash
# Run all voice service tests
npm test -- src/__tests__/services/

Manual Testing#

  1. Connection Test: Verify WebSocket connects successfully
  2. STT Test: Speak into microphone, verify transcription appears
  3. TTS Test: Send text, verify audio plays correctly
  4. Barge-In Test: Speak while AI is talking, verify AI stops
  5. Reconnection Test: Disable network, re-enable, verify reconnection
  6. Error Test: Send invalid data, verify error handling

Security Considerations#

Authentication#

typescript
// Include auth token in WebSocket connection
const webSocketClient = new WebSocketClient({
  baseUrl: `${WS_BASE_URL}?token=${authToken}`,
  // ...
});

// Include auth token in API calls
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  // Session will use authenticated user from token
});

Data Privacy#

  • Audio is streamed, not stored on client
  • Transcriptions are ephemeral (not persisted unless explicitly saved)
  • Session data includes minimal PII
  • All connections use WSS (WebSocket Secure) and HTTPS

Rate Limiting#

typescript
// Handle rate limit errors
voicePipelineService.on('error', (error) => {
  if (error.code === 'RATE_LIMIT_EXCEEDED') {
    // Show rate limit message
    // Wait before retrying
  }
});

Troubleshooting#

Common Issues#

Issue: WebSocket connection fails Solution: Check WS_BASE_URL configuration, verify network connectivity, check authentication token

Issue: Transcription is slow or inaccurate Solution: Check audio quality, adjust VAD threshold, use appropriate STT model size

Issue: TTS audio is choppy Solution: Increase chunk size, check network speed, enable audio buffering

Issue: Barge-in not working Solution: Verify bargeInConfig.enabled = true, check audio level threshold, ensure STT stream is active

Issue: Memory leak in long sessions Solution: Call dispose() when done, clear audio buffers regularly, use endSession() between conversations

Debug Logging#

typescript
// Enable debug logging
voicePipelineService.on('transcription:partial', (result) => {
  console.log('[STT Partial]', result.text, result.confidence);
});

voicePipelineService.on('tts:chunk', (chunk) => {
  console.log('[TTS Chunk]', chunk.chunkIndex, '/', chunk.totalChunks);
});

webSocketClient.on('messageSent', (message) => {
  console.log('[WS Sent]', message.type);
});

Future Enhancements#

  • Multi-language support with automatic language detection
  • Emotion detection from voice tone
  • Speaker diarization for multi-user conversations
  • Audio effects (reverb, EQ) for persona voice customization
  • Offline mode with local STT/TTS models
  • Advanced VAD with noise cancellation
  • Real-time translation between languages

References#