Eve library

OSHUN V1 Metis Generation And Tutoring SLOs

Spec

Revision 1

On this page

Status: Active

Date: April 10, 2026

Owners: OSHUN Platform, OSHUN Metis Engineering, OSHUN Safety

Related:

  • docs/releases/v1/reports/metis-v1-scorecard.md
  • docs/releases/v1/runbooks/metis-operator-workflows-runbook.md
  • docs/releases/v1/audits/metis/media-evaluation.md
  • docs/releases/v1/specs/todos.md

1. Purpose#

This document defines the canonical SLO layer for Metis generation and tutoring.

It exists so Metis release decisions are driven by explicit cost, latency, quality, and failure targets rather than generic uptime or ad hoc operator judgment.

These SLOs are the thresholds surfaced in the Metis admin monitoring workspace.

2. Scope#

This SLO set covers two launch-critical surfaces:

  1. generation Prerecorded lecture package generation, packaging, export, and release readiness for grounded educational outputs.
  2. tutoring Live and text tutoring runtime behavior, learner-facing response cadence, evaluation quality, and failure recovery.

3. Canonical Objectives#

SLO ID Surface Dimension Target Warning Critical Primary evidence
generation_cost generation cost average package cost <= $6.50 > $6.50 > $8.00 persisted lecture package complexity model
generation_latency generation latency P95 build latency <= 6000 ms > 6000 ms > 9000 ms build-job observability events
generation_quality generation quality average score >= 85/100 < 85 < 70 lecture media evaluation plus release-readiness fallback
generation_failure generation failure failure rate <= 5% > 5% > 10% blocked or failed build, export, and rights posture
tutoring_cost tutoring cost average session cost <= $1.50 > $1.50 > $2.50 persisted session estimated cost
tutoring_latency tutoring latency P95 tutor response <= 450 ms > 450 ms > 700 ms tutoring response-gap timing
tutoring_quality tutoring quality average score >= 85/100 < 85 < 70 tutor evaluation, embodiment evaluation, satisfaction fallback
tutoring_failure tutoring failure failure rate <= 4% > 4% > 8% escalated, abandoned, or stale tutoring sessions

4. Signal Mapping#

4.1 Generation#

  • generation_cost Derived from the persisted lecture package itself, using a deterministic complexity model over lessons, slides, diagrams, notes, locale variants, render jobs, and export bundles.
  • generation_latency Reads processingTimeMs from build-job observability events. When recent event data is absent, the admin workspace falls back to current build readiness evidence so the lane remains explicit rather than silently empty.
  • generation_quality Prefers persisted lecture media evaluation scores. When those are absent, it uses build-readiness evidence as a lower-fidelity fallback so release posture still degrades visibly.
  • generation_failure Combines explicit build or export event failures with derived blockers from review status, grounded lesson coverage, assessment support, and source-rights posture.

4.2 Tutoring#

  • tutoring_cost Uses persisted session estimated_cost from the real tutoring service.
  • tutoring_latency Uses learner-to-assistant message gaps and falls back to session duration per message when direct gaps are unavailable.
  • tutoring_quality Prefers canonical tutor evaluation reports and tutoring embodiment-quality evaluations, then falls back to learner satisfaction when no persisted evaluations exist yet.
  • tutoring_failure Counts escalated, abandoned, or stale active sessions as failures because these reflect real learner support breakdowns or recovery gaps.

5. Release Interpretation#

  • Any critical quality or failure SLO is release-blocking.
  • warning cost or latency SLOs require operator review and remediation ownership before release approval.
  • A release is not considered Metis-green if the SLO layer is unproven because evidence is missing; missing evidence must remain visible as degraded posture.

6. Operational Notes#

  • These thresholds are intentionally Metis-specific and should not be replaced by generic shell or Studio uptime targets.
  • If the cost model, quality suites, or tutoring latency instrumentation changes, this document and the admin monitoring workspace must change in the same task.
  • Any future per-model or per-course exception must be audit-backed and linked from the Metis incident or release-gate workspace.