Status: Active
Date: April 10, 2026
Owners: OSHUN Metis Engineering, OSHUN Psyche, OSHUN Yemaya, OSHUN Safety
Purpose#
V1-MET-096 adds canonical output-quality evaluation suites for:
- prerecorded lecture audio and video outputs
- caption and transcript coverage
- timing and sync posture across lecture assets
- live tutor speech and embodiment fidelity
The goal is not to pretend that Metis already has waveform-level DSP or pixel-level avatar telemetry in the service layer. The goal is to formalize the best launch-grade evaluation surface that the current Metis runtime actually supports, while aligning the scoring shape to existing quality baselines used in speech and embodied-media systems.
Existing Implementations Reviewed#
Before implementation, the current Metis seams were audited:
services/metis/src/metis/services/lecture_generation_service.pyservices/metis/src/metis/services/lecture_generation_helpers.pyservices/metis/src/metis/services/lecture_render_pipeline_helpers.pyservices/metis/src/metis/services/tutoring_service.pyservices/metis/src/metis/services/source_tutoring_helpers.py- existing grounded-evaluation patterns in
services/metis/src/metis/services/source_evaluation_service.py - existing tutor-evaluation patterns in
services/metis/src/metis/services/tutor_evaluation_service.py
Key conclusion:
- Metis already persists enough structured metadata to score media quality
deterministically:
- lesson script timing
- rendered slide and diagram outputs
- caption-track and transcript packaging
- live tutor fallback and fidelity posture
- interruption and recovery metadata
- teacher-representation approval and consent state
That means V1-MET-096 should be implemented as a first-class persisted
evaluation layer, not as one-off assertions in tests.
External Baselines Reviewed#
The implementation direction was informed by these primary references:
- ITU PESQ / objective speech-quality framing:
https://www.itu.int/dms_pubrec/itu-r/rec/m/R-REC-M.1079-2-200306-I!!PDF-E.pdf - SyncNet / lip-sync alignment framing:
https://arxiv.org/abs/1609.06773Note: Metis does not implement SyncNet itself here; it adopts the idea of explicit sync-quality scoring rather than leaving embodiment fidelity implicit. - Wav2Lip / embodied-media evaluation framing:
https://arxiv.org/abs/2008.10010
Inference from those sources:
- launch-grade Metis should separate speech quality, caption quality, and sync quality instead of collapsing them into one blended score
- embodiment quality should not only reflect runtime fidelity; it should also account for fallback posture, disclosure readiness, consent state, and interruption recovery
What Was Implemented#
V1-MET-096 adds:
- canonical media evaluation schemas in
services/metis/src/metis/schemas/media_evaluation.py - deterministic lecture and tutor evaluation helpers in
services/metis/src/metis/services/media_evaluation_helpers.py - persisted evaluation storage in
services/metis/src/metis/services/media_evaluation_service.py - lecture evaluation API surfaces on
/courses/{course_id}/lecture-package/evaluation - tutoring evaluation API surfaces on
/tutoring/sessions/{session_id}/quality-evaluation
Scoring Model#
Lecture evaluation scores:
- audio quality
- video quality
- caption quality
- sync quality
Tutor evaluation scores:
- speech quality
- sync quality
- embodiment fidelity
- recovery quality
Status thresholds:
pass:>= 85review:>= 70and< 85fail:< 70
Current Limitations#
This work intentionally does not claim more precision than the current runtime supports.
Not yet implemented here:
- waveform-level MOS/PESQ computation
- phoneme-to-viseme sync scoring from live avatar frames
- frame-accurate subtitle alignment against rendered media
Those require deeper Psyche or Yemaya telemetry than Metis currently stores. The
suite added in V1-MET-096 is the canonical launch-grade evaluation layer for
the persisted artifacts Metis does own today.