Eve library

OSHUN V1 Metis Media Evaluation Audit

Guide

Revision 1

On this page

Status: Active

Date: April 10, 2026

Owners: OSHUN Metis Engineering, OSHUN Psyche, OSHUN Yemaya, OSHUN Safety

Purpose#

V1-MET-096 adds canonical output-quality evaluation suites for:

  • prerecorded lecture audio and video outputs
  • caption and transcript coverage
  • timing and sync posture across lecture assets
  • live tutor speech and embodiment fidelity

The goal is not to pretend that Metis already has waveform-level DSP or pixel-level avatar telemetry in the service layer. The goal is to formalize the best launch-grade evaluation surface that the current Metis runtime actually supports, while aligning the scoring shape to existing quality baselines used in speech and embodied-media systems.

Existing Implementations Reviewed#

Before implementation, the current Metis seams were audited:

  • services/metis/src/metis/services/lecture_generation_service.py
  • services/metis/src/metis/services/lecture_generation_helpers.py
  • services/metis/src/metis/services/lecture_render_pipeline_helpers.py
  • services/metis/src/metis/services/tutoring_service.py
  • services/metis/src/metis/services/source_tutoring_helpers.py
  • existing grounded-evaluation patterns in services/metis/src/metis/services/source_evaluation_service.py
  • existing tutor-evaluation patterns in services/metis/src/metis/services/tutor_evaluation_service.py

Key conclusion:

  • Metis already persists enough structured metadata to score media quality deterministically:
    • lesson script timing
    • rendered slide and diagram outputs
    • caption-track and transcript packaging
    • live tutor fallback and fidelity posture
    • interruption and recovery metadata
    • teacher-representation approval and consent state

That means V1-MET-096 should be implemented as a first-class persisted evaluation layer, not as one-off assertions in tests.

External Baselines Reviewed#

The implementation direction was informed by these primary references:

  • ITU PESQ / objective speech-quality framing: https://www.itu.int/dms_pubrec/itu-r/rec/m/R-REC-M.1079-2-200306-I!!PDF-E.pdf
  • SyncNet / lip-sync alignment framing: https://arxiv.org/abs/1609.06773 Note: Metis does not implement SyncNet itself here; it adopts the idea of explicit sync-quality scoring rather than leaving embodiment fidelity implicit.
  • Wav2Lip / embodied-media evaluation framing: https://arxiv.org/abs/2008.10010

Inference from those sources:

  • launch-grade Metis should separate speech quality, caption quality, and sync quality instead of collapsing them into one blended score
  • embodiment quality should not only reflect runtime fidelity; it should also account for fallback posture, disclosure readiness, consent state, and interruption recovery

What Was Implemented#

V1-MET-096 adds:

  • canonical media evaluation schemas in services/metis/src/metis/schemas/media_evaluation.py
  • deterministic lecture and tutor evaluation helpers in services/metis/src/metis/services/media_evaluation_helpers.py
  • persisted evaluation storage in services/metis/src/metis/services/media_evaluation_service.py
  • lecture evaluation API surfaces on /courses/{course_id}/lecture-package/evaluation
  • tutoring evaluation API surfaces on /tutoring/sessions/{session_id}/quality-evaluation

Scoring Model#

Lecture evaluation scores:

  • audio quality
  • video quality
  • caption quality
  • sync quality

Tutor evaluation scores:

  • speech quality
  • sync quality
  • embodiment fidelity
  • recovery quality

Status thresholds:

  • pass: >= 85
  • review: >= 70 and < 85
  • fail: < 70

Current Limitations#

This work intentionally does not claim more precision than the current runtime supports.

Not yet implemented here:

  • waveform-level MOS/PESQ computation
  • phoneme-to-viseme sync scoring from live avatar frames
  • frame-accurate subtitle alignment against rendered media

Those require deeper Psyche or Yemaya telemetry than Metis currently stores. The suite added in V1-MET-096 is the canonical launch-grade evaluation layer for the persisted artifacts Metis does own today.