Decision#
Voice emotion: ONNX wav2vec2-emotion (6-class arousal/valence). Image emotion: ONNX HSEmotion (FER+ 7-class). Both run locally through the shared ML runtime; cloud APIs (AWS Comprehend Sentiment, Affectiva Affdex) are kept on hand for the long-tail "research-grade affect" flow but not used by default.
Emotion classification is inherently imprecise; we ship 6 / 7 discrete classes plus an arousal-valence continuous pair so callers can choose the granularity they need. We deliberately do NOT ship a "depression detector" or any clinical-grade affect model — the failure-mode cost is too high for what's a probabilistic signal.
Constraints#
- Privacy. Same as speaker-id (V1-P2-0214) — emotion signals are sensitive PII; default path is local.
- Latency. Voice emotion: ≤ 80 ms per 1 s of audio. Image emotion: ≤ 30 ms per face crop on WebGPU.
- Honest about uncertainty. Output must include per-class probability, not a single hard label.
Options#
Voice — ONNX wav2vec2-emotion (chosen)#
- HuggingFace
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dimexported to ONNX. Outputs three continuous dims (arousal / dominance / valence) + 6-class softmax (neutral / happy / angry / sad / fearful / disgusted). - ~310 MB FP16. We ship the INT8 quantised version (~80 MB) by default.
- Latency: ~50 ms per 1 s on WebGPU; ~140 ms WASM SIMD.
Image — HSEmotion ONNX (chosen)#
hsemotionrepository ships a small (~2 MB) MobileNet-V2 backbone trained on AffectNet + FER+. 7-class output (neutral / happy / surprise / sad / anger / disgust / fear / contempt).- Latency: ~5 ms per face crop on WebGPU.
- Pairs naturally with the face-detection step from the object-detection provider (V1-P2-0212).
Cloud — AWS Comprehend / Affectiva (kept, not default)#
- More classes, fine-grained intensity, in some cases (Affectiva) facial action unit (AU) decomposition.
- Useful when a research / enterprise flow needs auditable third-party affect labels.
Clinical-grade affect — rejected#
- Depression / bipolar / suicidal-ideation classification from voice or text is well-established as unreliable in the open literature. We refuse to ship this.
Output schema#
ts
export interface VoiceEmotionResult {
readonly arousal: number; // [-1, 1]
readonly valence: number; // [-1, 1]
readonly dominance: number; // [-1, 1]
readonly classes: Readonly<
Record<
'neutral' | 'happy' | 'angry' | 'sad' | 'fearful' | 'disgusted',
number /* probability in [0,1] */
>
>;
}
export interface ImageEmotionResult {
readonly classes: Readonly<
Record<
| 'neutral'
| 'happy'
| 'surprise'
| 'sad'
| 'anger'
| 'disgust'
| 'fear'
| 'contempt',
number
>
>;
}
Action items#
- Pull
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dimand export to ONNX; quantise to INT8. - Pull
hsemotion-onnx(MobileNet-V2 backbone); check FP16 + INT8 variants into the registry. - Wire
libs/shared/src/ml/emotion.tsnamespace withvoiceEmotion(audio)andfaceEmotion(faceCrop)entrypoints. - Add
disclaimerflag to the API surface so callers programmatically receive a "probabilistic signal — do not use for clinical decisions" note.
References: