# Speaker Identification Provider — V1-P2-0214

## Decision

**Use ONNX ECAPA-TDNN (192-dim embedding) as the default speaker-id model.
Reserve cloud APIs (Pindrop, AWS Voice ID) for high-stakes authentication where
a fraud-grade verification network behind the embedding is required. Reject
Resemblyzer.**

ECAPA-TDNN runs locally through the shared ML runtime, embeddings are stored in
the user's local Iris memory partition, and matching is done client-side via
cosine similarity. The cloud path is gated behind explicit user opt-in for any
flow that authorises a financial / identity action on the speaker's voice alone.

---

## Constraints

1. **Privacy.** Voiceprints are biometric — they cannot leave the device for the
   default flow.
2. **Embedding stability across sessions.** Cosine distance between two
   utterances of the same speaker must drop below 0.25 with ≥ 90% recall.
3. **Diarisation friendly.** Embedding extraction must be fast enough (≤ 50 ms
   per 1 s of audio) that we can extract per-segment embeddings during a
   multi-speaker conversation without falling behind real-time playback.

---

## Options considered

### A. ONNX ECAPA-TDNN (chosen)

- 192-dim embedding, ~17 MB FP16. Inference: ~25 ms per 1 s of audio on WebGPU,
  ~80 ms on WASM SIMD.
- Best EER (~0.86%) of any portable speaker-id model on VoxCeleb1-O.
- Trained on VoxCeleb1+2 with multilingual augmentation; generalises to
  non-English speakers without fine-tuning.
- ONNX export from speechbrain (`spkrec-ecapa-voxceleb`) is the industry
  reference — verified parity to the PyTorch original.

### B. Resemblyzer (rejected)

- Older GE2E-style 256-dim embedding. EER ≈ 6%, several × worse than ECAPA-TDNN
  on the same eval.
- No first-class ONNX export; runs through PyTorch only.

### C. Cloud APIs (kept for high-stakes auth)

- Pindrop / AWS Voice ID / Azure Speaker Recognition include liveness detection,
  deepfake-resistance, and cross-channel voiceprint matching. Useful for "voice
  is the second factor in a financial login" flows.
- Per-verification billing, so cost-bounded.
- Privacy gate: explicit opt-in, voiceprint stored in the cloud provider's
  enclave, not in our database.

---

## Output schema

```ts
export interface SpeakerEmbedding {
  /** 192-dim L2-normalised float vector. */
  readonly vector: Float32Array;
  /** Source clip duration in ms (for confidence weighting). */
  readonly clipDurationMs: number;
  /** Embedding model + version, for compat checks. */
  readonly model: 'ecapa-tdnn-v1';
}

export function speakerSimilarity(
  a: SpeakerEmbedding,
  b: SpeakerEmbedding
): number; // cosine in [-1,1]
```

The same `SpeakerEmbedding` shape is produced by the cloud path so downstream
code can mix-and-match where the voiceprint store allows.

---

## Action items

- [ ] Pull `spkrec-ecapa-voxceleb` from speechbrain; export to ONNX via
      `torch.onnx.export`; verify EER on VoxCeleb1-O within ±0.1% of the
      published number.
- [ ] Quantise to FP16 + INT8; check both into the registry with SHA-256.
- [ ] Wire the `libs/shared/src/ml/speaker-id.ts` namespace.
- [ ] Implement liveness gate: reject embeddings extracted from clips < 0.5 s or
      with VAD-detected silence > 80%.

References:

- [SpeechBrain ECAPA-TDNN](https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb)
- [VoxCeleb eval](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/)
- [Pindrop Voice ID](https://www.pindrop.com/)
