# OSHUN V1 Metis Media Evaluation Audit

Status: Active

Date: April 10, 2026

Owners: OSHUN Metis Engineering, OSHUN Psyche, OSHUN Yemaya, OSHUN Safety

## Purpose

`V1-MET-096` adds canonical output-quality evaluation suites for:

- prerecorded lecture audio and video outputs
- caption and transcript coverage
- timing and sync posture across lecture assets
- live tutor speech and embodiment fidelity

The goal is not to pretend that Metis already has waveform-level DSP or
pixel-level avatar telemetry in the service layer. The goal is to formalize the
best launch-grade evaluation surface that the current Metis runtime actually
supports, while aligning the scoring shape to existing quality baselines used in
speech and embodied-media systems.

## Existing Implementations Reviewed

Before implementation, the current Metis seams were audited:

- `services/metis/src/metis/services/lecture_generation_service.py`
- `services/metis/src/metis/services/lecture_generation_helpers.py`
- `services/metis/src/metis/services/lecture_render_pipeline_helpers.py`
- `services/metis/src/metis/services/tutoring_service.py`
- `services/metis/src/metis/services/source_tutoring_helpers.py`
- existing grounded-evaluation patterns in
  `services/metis/src/metis/services/source_evaluation_service.py`
- existing tutor-evaluation patterns in
  `services/metis/src/metis/services/tutor_evaluation_service.py`

Key conclusion:

- Metis already persists enough structured metadata to score media quality
  deterministically:
  - lesson script timing
  - rendered slide and diagram outputs
  - caption-track and transcript packaging
  - live tutor fallback and fidelity posture
  - interruption and recovery metadata
  - teacher-representation approval and consent state

That means `V1-MET-096` should be implemented as a first-class persisted
evaluation layer, not as one-off assertions in tests.

## External Baselines Reviewed

The implementation direction was informed by these primary references:

- ITU PESQ / objective speech-quality framing:
  `https://www.itu.int/dms_pubrec/itu-r/rec/m/R-REC-M.1079-2-200306-I!!PDF-E.pdf`
- SyncNet / lip-sync alignment framing: `https://arxiv.org/abs/1609.06773` Note:
  Metis does not implement SyncNet itself here; it adopts the idea of explicit
  sync-quality scoring rather than leaving embodiment fidelity implicit.
- Wav2Lip / embodied-media evaluation framing:
  `https://arxiv.org/abs/2008.10010`

Inference from those sources:

- launch-grade Metis should separate speech quality, caption quality, and sync
  quality instead of collapsing them into one blended score
- embodiment quality should not only reflect runtime fidelity; it should also
  account for fallback posture, disclosure readiness, consent state, and
  interruption recovery

## What Was Implemented

`V1-MET-096` adds:

- canonical media evaluation schemas in
  `services/metis/src/metis/schemas/media_evaluation.py`
- deterministic lecture and tutor evaluation helpers in
  `services/metis/src/metis/services/media_evaluation_helpers.py`
- persisted evaluation storage in
  `services/metis/src/metis/services/media_evaluation_service.py`
- lecture evaluation API surfaces on
  `/courses/{course_id}/lecture-package/evaluation`
- tutoring evaluation API surfaces on
  `/tutoring/sessions/{session_id}/quality-evaluation`

## Scoring Model

Lecture evaluation scores:

- audio quality
- video quality
- caption quality
- sync quality

Tutor evaluation scores:

- speech quality
- sync quality
- embodiment fidelity
- recovery quality

Status thresholds:

- `pass`: `>= 85`
- `review`: `>= 70` and `< 85`
- `fail`: `< 70`

## Current Limitations

This work intentionally does not claim more precision than the current runtime
supports.

Not yet implemented here:

- waveform-level MOS/PESQ computation
- phoneme-to-viseme sync scoring from live avatar frames
- frame-accurate subtitle alignment against rendered media

Those require deeper Psyche or Yemaya telemetry than Metis currently stores. The
suite added in `V1-MET-096` is the canonical launch-grade evaluation layer for
the persisted artifacts Metis does own today.
