# Voice Chat Architecture

## Overview

The Lilith mobile app implements a complete real-time voice chat system with streaming Speech-to-Text (STT), bidirectional conversation with AI personas, streaming Text-to-Speech (TTS), and intelligent barge-in capabilities.

## Architecture Diagram

```
┌──────────────┐        WebSocket         ┌──────────────┐
│   Mobile     │◄─────────────────────────►│  Voice       │
│   Client     │                           │  Pipeline    │
│              │        HTTPS/REST         │  Service     │
│              │◄─────────────────────────►│              │
└──────────────┘                           └──────────────┘
       │                                           │
       │ Audio Capture                             │
       │ Audio Playback                            │
       │                                           │
       ▼                                           ▼
┌──────────────┐                           ┌──────────────┐
│  React       │                           │   Backend    │
│  Native      │                           │   Services   │
│  Audio API   │                           │              │
└──────────────┘                           │  - STT       │
                                            │  - LLM       │
                                            │  - TTS       │
                                            └──────────────┘
```

## Core Components

### 1. WebSocketClient (`src/services/WebSocketClient.ts`)

**Purpose**: Provides reliable WebSocket connectivity with automatic reconnection, message queuing, and stream management.

**Key Features**:
- **Automatic Reconnection**: Exponential backoff with stability tracking
- **Message Queuing**: Queues messages when disconnected, sends when reconnected
- **Stream Abstraction**: Creates bidirectional streams for STT/TTS
- **Request/Response Pattern**: Async communication with timeout handling
- **Ping/Pong Keep-Alive**: Heartbeat mechanism to maintain connections

**Configuration**:
```typescript
const wsConfig = {
  baseUrl: 'wss://api.lilith.app/ws',
  reconnect: true,
  reconnectInterval: 1000,      // Base delay in ms
  reconnectAttempts: 5,          // Max reconnection attempts
  timeout: 30000,                // Request timeout in ms
  pingInterval: 30000,           // Heartbeat interval in ms
};
```

**API**:
```typescript
// Connect to WebSocket server
await webSocketClient.connect();

// Send a message
webSocketClient.send({
  type: 'message',
  data: { content: 'Hello' }
});

// Request/response pattern
const response = await webSocketClient.request({
  type: 'query',
  data: { query: 'status' }
});

// Create a streaming connection
const stream = webSocketClient.createStream({
  endpoint: '/v1/stt/stream',
  params: { language: 'en-US' }
});

// Handle stream messages
stream.onmessage = (event) => {
  console.log('Received:', event.data);
};

// Send data to stream
stream.send({ audio: audioData });

// Close stream
stream.close();

// Disconnect
await webSocketClient.disconnect();
```

**Events**:
- `connected`: WebSocket connection established
- `disconnected`: WebSocket connection closed
- `connecting`: Attempting to connect
- `error`: Error occurred
- `reconnecting`: Attempting to reconnect (includes attempt number, max attempts, delay)
- `reconnectFailed`: Max reconnection attempts reached
- `message`: General message received
- `streamMessage`: Stream-specific message received
- `streamError`: Stream error occurred
- `pong`: Pong response to ping received
- `messageSent`: Message successfully sent

**Reconnection Logic**:
```typescript
// Connection stability tracking prevents infinite reconnect loops
// Only resets attempt counter after connection proves stable (1 second)
private scheduleReconnect(): void {
  this.isInReconnectCycle = true;
  this.reconnectAttempt++;
  const delay = this.config.reconnectInterval * Math.pow(2, this.reconnectAttempt - 1);

  setTimeout(() => this.connect(), delay);
}
```

**Test Coverage**: 27/27 tests passing (100%)
- Connection management
- Message handling
- Request/response pattern
- Streaming interface
- Reconnection with exponential backoff
- Ping/pong keep-alive
- Error handling
- Resource cleanup

### 2. VoicePipelineService (`src/services/VoicePipelineService.ts`)

**Purpose**: Orchestrates end-to-end voice conversations (STT → LLM → TTS) with session management and barge-in support.

**Key Features**:
- **Session Management**: Create and manage voice conversation sessions
- **STT Streaming**: Real-time speech-to-text transcription with VAD
- **TTS Streaming**: Chunked audio synthesis with progress tracking
- **Barge-In**: Interrupt AI speech when user starts speaking
- **Connection Lifecycle**: Automatic initialization and cleanup

**Session Types**:
```typescript
type SessionMode = 'push-to-talk' | 'vad' | 'hybrid';

interface VoiceSession {
  id: string;
  userId: string;
  personaId?: string;
  mode: SessionMode;
  vadConfig?: {
    threshold: number;        // 0.0-1.0, default 0.7
    silenceTimeout: number;   // ms, default 1500
  };
  bargeInConfig?: {
    enabled: boolean;
    threshold: number;        // 0.0-1.0, default 0.8
    fadeOutDuration: number;  // ms, default 200
  };
  status: 'active' | 'paused' | 'completed';
}
```

**API**:
```typescript
// Initialize service
await voicePipelineService.initialize();

// Create a voice session
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  personaId: 'zen-master',
  mode: 'hybrid',
  vadConfig: {
    threshold: 0.7,
    silenceTimeout: 1500
  },
  bargeInConfig: {
    enabled: true,
    threshold: 0.8,
    fadeOutDuration: 200
  }
});

// Start STT streaming
const sttStream = voicePipelineService.startSTTStream({
  language: 'en-US',
  model: 'whisper-small',
  enablePartialResults: true,
  enableVAD: true
});

// Send audio to STT
sttStream.sendAudio(audioBufferData);

// Listen for transcriptions
voicePipelineService.on('transcription:partial', (result) => {
  console.log('Partial:', result.text);
});

voicePipelineService.on('transcription:final', (result) => {
  console.log('Final:', result.text, 'Confidence:', result.confidence);
});

// Stop STT
sttStream.stop();

// Start TTS streaming
const ttsStream = voicePipelineService.startTTSStream(
  'Hello, how can I help you today?',
  {
    voiceId: 'voice-zen-master',
    language: 'en-US',
    speed: 1.0,
    pitch: 1.0,
    format: 'opus'
  }
);

// Handle TTS chunks
ttsStream.onChunk((chunk) => {
  playAudio(chunk.audio);
  console.log('Progress:', chunk.progress);
});

// Handle completion
ttsStream.onComplete(() => {
  console.log('TTS complete');
});

// Trigger barge-in
await voicePipelineService.triggerBargeIn(audioLevel);

// End session
await voicePipelineService.endSession();

// Cleanup
await voicePipelineService.dispose();
```

**Events**:
- `initialized`: Service initialized
- `connected`: WebSocket connected
- `disconnected`: WebSocket disconnected
- `sessionCreated`: Voice session created
- `sessionEnded`: Voice session ended
- `transcription:partial`: Partial transcription result
- `transcription:final`: Final transcription result
- `vad:activity`: Voice activity detected (active, audioLevel, timestamp)
- `stt:error`: STT streaming error
- `stt:stopped`: STT streaming stopped
- `tts:chunk`: TTS audio chunk received
- `tts:complete`: TTS streaming complete
- `tts:error`: TTS streaming error
- `tts:cancelled`: TTS streaming cancelled
- `bargeIn:triggered`: Barge-in event triggered (interrupted, timestamp, audioLevel)
- `error`: General error
- `reconnecting`: WebSocket reconnecting
- `reconnectFailed`: WebSocket reconnection failed

**Barge-In Logic**:
```typescript
// When barge-in is triggered:
// 1. API call to backend to validate and log barge-in event
// 2. If successful, stops all active TTS streams
// 3. Emits bargeIn:triggered event with details

async triggerBargeIn(audioLevel: number): Promise<void> {
  const response = await fetch(
    `${this.config.apiBaseUrl}/v1/voice/sessions/${session.id}/barge-in`,
    {
      method: 'POST',
      body: JSON.stringify({ audioLevel, timestamp: Date.now() })
    }
  );

  const result: BargeInEvent = await response.json();

  if (result.interrupted) {
    // Stop all active TTS streams
    this.activeStreams.forEach((stream, streamId) => {
      if (streamId.startsWith('tts-')) {
        stream.close();
        this.activeStreams.delete(streamId);
      }
    });
  }
}
```

**Test Coverage**: 22/22 tests passing (100%)
- Service initialization
- Session creation and management
- STT streaming (transcription, VAD)
- TTS streaming (chunks, completion)
- Barge-in functionality
- Connection status
- Error handling
- Resource cleanup

### 3. Service Integration (`src/services/serviceInstances.ts`)

**Purpose**: Centralized service initialization with proper configuration and dependency injection.

**Configuration**:
```typescript
const API_BASE_URL = process.env.REACT_APP_API_BASE_URL || 'https://api.lilith.app';
const WS_BASE_URL = process.env.REACT_APP_WS_BASE_URL || 'wss://api.lilith.app/ws';

export const webSocketClient = new WebSocketClient({
  baseUrl: WS_BASE_URL,
  reconnect: true,
  reconnectInterval: 1000,
  reconnectAttempts: 5,
  timeout: 30000,
  pingInterval: 30000,
});

export const voicePipelineService = new VoicePipelineService({
  apiBaseUrl: API_BASE_URL,
  wsBaseUrl: WS_BASE_URL,
  sttEndpoint: '/v1/stt/transcribe',
  ttsEndpoint: '/v1/tts/synthesize',
  voiceSessionEndpoint: '/v1/voice/sessions',
  webSocketConfig: {
    reconnect: true,
    reconnectInterval: 1000,
    reconnectAttempts: 5,
    timeout: 30000,
    pingInterval: 30000,
  },
});
```

## Usage Examples

### Basic Voice Chat Flow

```typescript
import { voicePipelineService } from '../services/serviceInstances';
import { AudioRecorder } from 'react-native-audio-recorder-player';

const audioRecorder = new AudioRecorder();

// 1. Initialize and create session
await voicePipelineService.initialize();
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  personaId: 'zen-master',
  mode: 'hybrid'
});

// 2. Start recording and STT streaming
const sttStream = voicePipelineService.startSTTStream();

audioRecorder.startRecording();
audioRecorder.addRecordBackListener((e) => {
  // Send audio chunks to STT
  sttStream.sendAudio(e.currentMetering);
});

// 3. Listen for transcription
voicePipelineService.on('transcription:final', async (result) => {
  console.log('User said:', result.text);

  // Stop recording
  await audioRecorder.stopRecording();
  sttStream.stop();

  // Get AI response (via separate LLM API call)
  const aiResponse = await fetch(`${API_BASE_URL}/v1/chat`, {
    method: 'POST',
    body: JSON.stringify({
      message: result.text,
      personaId: 'zen-master'
    })
  }).then(r => r.json());

  // 4. Start TTS playback
  const ttsStream = voicePipelineService.startTTSStream(aiResponse.text);

  const audioPlayer = new AudioRecorder();
  const audioChunks: ArrayBuffer[] = [];

  ttsStream.onChunk((chunk) => {
    audioChunks.push(chunk.audio);
  });

  ttsStream.onComplete(() => {
    // Combine chunks and play
    const fullAudio = combineAudioChunks(audioChunks);
    audioPlayer.startPlayer(fullAudio);
  });
});

// 5. Handle barge-in
audioRecorder.addRecordBackListener((e) => {
  if (e.currentMetering > -30) { // User is speaking
    voicePipelineService.triggerBargeIn(e.currentMetering);
    // Restart recording flow
  }
});

// 6. Cleanup
await voicePipelineService.endSession();
await voicePipelineService.dispose();
```

### Push-to-Talk Mode

```typescript
// Use push-to-talk mode for simpler interaction
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  mode: 'push-to-talk'  // No VAD, user controls recording
});

// User presses button
function onPressStart() {
  const sttStream = voicePipelineService.startSTTStream({
    enableVAD: false  // Disable VAD in push-to-talk
  });

  audioRecorder.startRecording();
  // ... send audio to sttStream
}

// User releases button
function onPressEnd() {
  audioRecorder.stopRecording();
  sttStream.stop();
}
```

### VAD (Voice Activity Detection) Mode

```typescript
// Use VAD mode for hands-free operation
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  mode: 'vad',
  vadConfig: {
    threshold: 0.7,      // Sensitivity (0.0 = any sound, 1.0 = only loud sounds)
    silenceTimeout: 1500  // ms of silence before ending transcription
  }
});

// Listen for VAD events
voicePipelineService.on('vad:activity', (event) => {
  if (event.active) {
    console.log('User started speaking');
    // Show speaking indicator
  } else {
    console.log('User stopped speaking');
    // Hide speaking indicator
  }
});
```

## Error Handling

### Connection Errors

```typescript
voicePipelineService.on('error', (error) => {
  console.error('Service error:', error);
  // Show error UI
});

voicePipelineService.on('reconnecting', (info) => {
  console.log(`Reconnecting attempt ${info.attempt}/${info.maxAttempts}`);
  // Show reconnecting indicator
});

voicePipelineService.on('reconnectFailed', () => {
  console.error('Failed to reconnect');
  // Show offline mode or retry button
});
```

### Stream Errors

```typescript
// STT errors
voicePipelineService.on('stt:error', (error) => {
  console.error('STT error:', error);
  // Restart STT stream or show error
});

// TTS errors
const ttsStream = voicePipelineService.startTTSStream(text);
ttsStream.onError((error) => {
  console.error('TTS error:', error);
  // Retry or show error
});
```

## Performance Considerations

### Audio Chunk Size
```typescript
// Optimize chunk size for network conditions
const STT_CHUNK_SIZE = 4096;  // bytes
const TTS_CHUNK_SIZE = 4096;  // bytes

// Smaller chunks = more real-time, more overhead
// Larger chunks = less real-time, less overhead
```

### Memory Management
```typescript
// Clear audio buffers when done
audioChunks = [];

// Dispose services when component unmounts
useEffect(() => {
  return () => {
    voicePipelineService.dispose();
  };
}, []);
```

### Network Optimization
```typescript
// Use opus format for smaller payload size
const ttsStream = voicePipelineService.startTTSStream(text, {
  format: 'opus'  // ~50% smaller than wav
});

// Enable compression in STT
const sttStream = voicePipelineService.startSTTStream({
  enablePartialResults: true  // Get results faster
});
```

## Testing

### Unit Tests

```bash
# Run WebSocketClient tests
npm test -- src/__tests__/services/WebSocketClient.test.ts

# Run VoicePipelineService tests
npm test -- src/__tests__/services/VoicePipelineService.test.ts
```

### Integration Tests

```bash
# Run all voice service tests
npm test -- src/__tests__/services/
```

### Manual Testing

1. **Connection Test**: Verify WebSocket connects successfully
2. **STT Test**: Speak into microphone, verify transcription appears
3. **TTS Test**: Send text, verify audio plays correctly
4. **Barge-In Test**: Speak while AI is talking, verify AI stops
5. **Reconnection Test**: Disable network, re-enable, verify reconnection
6. **Error Test**: Send invalid data, verify error handling

## Security Considerations

### Authentication

```typescript
// Include auth token in WebSocket connection
const webSocketClient = new WebSocketClient({
  baseUrl: `${WS_BASE_URL}?token=${authToken}`,
  // ...
});

// Include auth token in API calls
const session = await voicePipelineService.createSession({
  userId: 'user-123',
  // Session will use authenticated user from token
});
```

### Data Privacy

- Audio is streamed, not stored on client
- Transcriptions are ephemeral (not persisted unless explicitly saved)
- Session data includes minimal PII
- All connections use WSS (WebSocket Secure) and HTTPS

### Rate Limiting

```typescript
// Handle rate limit errors
voicePipelineService.on('error', (error) => {
  if (error.code === 'RATE_LIMIT_EXCEEDED') {
    // Show rate limit message
    // Wait before retrying
  }
});
```

## Troubleshooting

### Common Issues

**Issue**: WebSocket connection fails
**Solution**: Check `WS_BASE_URL` configuration, verify network connectivity, check authentication token

**Issue**: Transcription is slow or inaccurate
**Solution**: Check audio quality, adjust VAD threshold, use appropriate STT model size

**Issue**: TTS audio is choppy
**Solution**: Increase chunk size, check network speed, enable audio buffering

**Issue**: Barge-in not working
**Solution**: Verify `bargeInConfig.enabled = true`, check audio level threshold, ensure STT stream is active

**Issue**: Memory leak in long sessions
**Solution**: Call `dispose()` when done, clear audio buffers regularly, use `endSession()` between conversations

### Debug Logging

```typescript
// Enable debug logging
voicePipelineService.on('transcription:partial', (result) => {
  console.log('[STT Partial]', result.text, result.confidence);
});

voicePipelineService.on('tts:chunk', (chunk) => {
  console.log('[TTS Chunk]', chunk.chunkIndex, '/', chunk.totalChunks);
});

webSocketClient.on('messageSent', (message) => {
  console.log('[WS Sent]', message.type);
});
```

## Future Enhancements

- Multi-language support with automatic language detection
- Emotion detection from voice tone
- Speaker diarization for multi-user conversations
- Audio effects (reverb, EQ) for persona voice customization
- Offline mode with local STT/TTS models
- Advanced VAD with noise cancellation
- Real-time translation between languages

## References

- [WebSocket API Documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket)
- [React Native Audio Recorder Player](https://github.com/hyochan/react-native-audio-recorder-player)
- [Opus Audio Codec](https://opus-codec.org/)
- [Voice Activity Detection](https://en.wikipedia.org/wiki/Voice_activity_detection)
