# ASR / VAD Provider — V1-P2-0213

## Decision

**Default ASR: ONNX Whisper (`tiny.en`, `base.en`, `small.en`, `medium.en`)
loaded through the shared ML runtime, with INT8 / FP16 quantisation per device
tier. Default VAD: ONNX Silero VAD (v5). Reserve Deepgram (Nova-3) as the cloud
fallback for >5-minute streaming transcription where Whisper-medium cannot keep
up on the device.**

VAD always runs locally — sending raw audio to a cloud just to detect silence
wastes bytes and bandwidth. ASR routes through the local model by default; the
cloud path is opt-in per call.

---

## Constraints

1. **Privacy-first.** Voice should be transcribed locally for the default flow;
   no audio leaves the device unless the user explicitly opts into cloud-grade
   transcription.
2. **Streaming partial transcripts.** First word must surface in < 400 ms after
   the speaker pauses; partial updates every 200 ms.
3. **Multilingual reach.** At minimum the major Latin-script languages
   (en/es/fr/de/it/pt) and CJK (zh/ja/ko); cloud fallback handles long-tail
   languages.
4. **VAD recall ≥ 0.95 at < 1% false-trigger rate** in noisy environments
   (fitness app, café).

---

## Options considered

### ASR

#### A. ONNX Whisper (chosen)

- OpenAI's Whisper has the best zero-shot multilingual performance of any open
  ASR model. ONNX exports from `transformers.js` and `whisper-cpp` are both
  validated.
- Variant landscape: `tiny.en` (~39 MB FP16, real-time on CPU), `base.en` (~74
  MB), `small.en` (~244 MB; default mid-tier), `medium.en` (~769 MB; opt-in for
  accuracy).
- Beam search is slow in JS; we ship the Rust `ml-postprocess` kernel for greedy
  decoding + length-normalised log-prob ranking which matches `transformers`
  reference output to within 1 word per 1000.

#### B. Deepgram Nova-3 (cloud fallback, kept)

- Best WER on long-form English (~3.0 vs Whisper-medium's ~5.5).
- Streaming WebSocket API with sub-300 ms partials.
- Per-minute billing — fine for cloud-tier usage gated behind opt-in.

#### C. Distil-Whisper (deferred)

- Faster-than-Whisper on the same architecture but the published ONNX exports
  are larger than the equivalent Whisper variant for the same WER. Revisit at
  V2.

### VAD

#### D. ONNX Silero VAD v5 (chosen)

- 1.7 MB ONNX, runs in < 1 ms per 30 ms audio chunk on CPU.
- Recall 0.97 / false-trigger 0.4% on the Silero validation suite — comfortably
  inside our envelope.
- Stateful (RNN-based); shared loader exposes a `VadStream` helper that
  maintains the hidden state across chunks.

#### E. WebRTC VAD — rejected

- Energy-based; brittle in non-speech-but-stationary noise (HVAC, vehicle
  interior). Recall drops to ~0.85 at low false-trigger.

#### F. Cloud VAD — rejected

- A round trip per 30 ms chunk is silly; no provider offers it this granular
  anyway.

---

## Latency budget

| component                          | target (p50) | target (p99) |
| ---------------------------------- | -----------: | -----------: |
| Silero VAD per 30 ms chunk         |       0.3 ms |       1.0 ms |
| Whisper-tiny.en short clip (<5s)   |       180 ms |       400 ms |
| Whisper-base.en short clip (<5s)   |       320 ms |       650 ms |
| Whisper-small.en short clip (<5s)  |       550 ms |     1 100 ms |
| Whisper-medium.en short clip (<5s) |     1 600 ms |     2 800 ms |
| Deepgram Nova-3 streaming partial  |       150 ms |       300 ms |

(All numbers measured on a Macbook Air M2; WebGPU + INT8 on Whisper.)

---

## Action items

- [ ] Pull Whisper ONNX exports (`tiny.en`/`base.en`/`small.en`/ `medium.en`)
      from `xenova/whisper-*`; verify WER on the LibriSpeech `test-clean` set
      within ±0.3 of the PyTorch ref.
- [ ] Pull Silero VAD v5 ONNX; verify recall + false-trigger on the Silero
      validation suite.
- [ ] Implement a Rust greedy decoder for Whisper at
      `libs/shared/crates/ml-postprocess/src/whisper.rs`.
- [ ] Wire the `libs/shared/src/ml/asr.ts` and `vad.ts` namespaces.
- [ ] Add a Deepgram client at `libs/aje/agents/src/asr/deepgram.ts` (cloud-tier
      opt-in path).

References:

- [transformers.js Whisper](https://huggingface.co/Xenova/whisper-base.en)
- [Silero VAD](https://github.com/snakers4/silero-vad)
- [Deepgram Nova-3](https://developers.deepgram.com/docs/models-overview)
