Decision#
Use ONNX ECAPA-TDNN (192-dim embedding) as the default speaker-id model. Reserve cloud APIs (Pindrop, AWS Voice ID) for high-stakes authentication where a fraud-grade verification network behind the embedding is required. Reject Resemblyzer.
ECAPA-TDNN runs locally through the shared ML runtime, embeddings are stored in the user's local Iris memory partition, and matching is done client-side via cosine similarity. The cloud path is gated behind explicit user opt-in for any flow that authorises a financial / identity action on the speaker's voice alone.
Constraints#
- Privacy. Voiceprints are biometric — they cannot leave the device for the default flow.
- Embedding stability across sessions. Cosine distance between two utterances of the same speaker must drop below 0.25 with ≥ 90% recall.
- Diarisation friendly. Embedding extraction must be fast enough (≤ 50 ms per 1 s of audio) that we can extract per-segment embeddings during a multi-speaker conversation without falling behind real-time playback.
Options considered#
A. ONNX ECAPA-TDNN (chosen)#
- 192-dim embedding, ~17 MB FP16. Inference: ~25 ms per 1 s of audio on WebGPU, ~80 ms on WASM SIMD.
- Best EER (~0.86%) of any portable speaker-id model on VoxCeleb1-O.
- Trained on VoxCeleb1+2 with multilingual augmentation; generalises to non-English speakers without fine-tuning.
- ONNX export from speechbrain (
spkrec-ecapa-voxceleb) is the industry reference — verified parity to the PyTorch original.
B. Resemblyzer (rejected)#
- Older GE2E-style 256-dim embedding. EER ≈ 6%, several × worse than ECAPA-TDNN on the same eval.
- No first-class ONNX export; runs through PyTorch only.
C. Cloud APIs (kept for high-stakes auth)#
- Pindrop / AWS Voice ID / Azure Speaker Recognition include liveness detection, deepfake-resistance, and cross-channel voiceprint matching. Useful for "voice is the second factor in a financial login" flows.
- Per-verification billing, so cost-bounded.
- Privacy gate: explicit opt-in, voiceprint stored in the cloud provider's enclave, not in our database.
Output schema#
ts
export interface SpeakerEmbedding {
/** 192-dim L2-normalised float vector. */
readonly vector: Float32Array;
/** Source clip duration in ms (for confidence weighting). */
readonly clipDurationMs: number;
/** Embedding model + version, for compat checks. */
readonly model: 'ecapa-tdnn-v1';
}
export function speakerSimilarity(
a: SpeakerEmbedding,
b: SpeakerEmbedding
): number; // cosine in [-1,1]
The same SpeakerEmbedding shape is produced by the cloud path so downstream
code can mix-and-match where the voiceprint store allows.
Action items#
- Pull
spkrec-ecapa-voxcelebfrom speechbrain; export to ONNX viatorch.onnx.export; verify EER on VoxCeleb1-O within ±0.1% of the published number. - Quantise to FP16 + INT8; check both into the registry with SHA-256.
- Wire the
libs/shared/src/ml/speaker-id.tsnamespace. - Implement liveness gate: reject embeddings extracted from clips < 0.5 s or with VAD-detected silence > 80%.
References: