Eve library

OSHUN V1 Metis Operator Workflows Runbook

Runbook

Revision 1

On this page

Status: Active

Date: April 10, 2026

Owners: OSHUN Metis Product, OSHUN Metis Admin Engineering, OSHUN Safety, OSHUN Support, OSHUN Program Operations

Related:

  • docs/releases/v1/reports/metis-v1-scorecard.md
  • docs/releases/v1/reports/metis-v1-workstream-tracking.md
  • docs/releases/v1/reports/metis-subsystem-ownership-matrix.md
  • docs/releases/v1/specs/todos.md
  • apps/metis/admin/src/app/page.tsx
  • apps/metis/admin/src/app/reviews/page.tsx
  • apps/metis/admin/src/app/rights/page.tsx
  • apps/metis/admin/src/app/content/page.tsx
  • apps/metis/admin/src/app/complaints/page.tsx
  • apps/metis/admin/src/app/users/page.tsx
  • apps/metis/admin/src/app/monitoring/page.tsx
  • apps/metis/admin/src/app/incidents/page.tsx
  • apps/metis/admin/src/app/audit/page.tsx

1. Purpose#

This document is the canonical runbook and training guide for Metis operator workflows in OSHUN V1.

It exists to answer five operating questions in one place:

  1. Which Metis admin surfaces are considered launch-critical operator tooling?
  2. Which operator role owns each queue, lane, or escalation path?
  3. What is the correct step-by-step flow for review, rights, moderation, support, learner operations, monitoring, and incident command?
  4. What training and signoff must an operator complete before working without supervision?
  5. What evidence must be present before an operator closes, escalates, or approves a Metis action?

This is a Metis-specific execution artifact, not a generic Studio operations document.

2. Operator Surface Map#

Surface Route Primary users Core purpose Must produce
Dashboard / shift lead, incident commander start-of-shift posture, quick routing, queue pressure review queue routing decision
Reviews /reviews curriculum reviewer, safety reviewer approve or reject publication packages, source bundles, prompt/runtime artifacts, avatar/voice artifacts review decision with rationale
Source Rights /rights rights reviewer, compliance reviewer inspect BYOM provenance, license posture, and allowed-use restrictions rights decision or blocker
Content /content moderation operator, safety reviewer inspect generated educational content and tutor abuse incidents moderation decision
Complaints /complaints learner-support operator triage content, tutor, grading, and export complaints acknowledgement, investigation, escalation, or resolution
Users /users and /users/[userId] learner-support lead pause/resume progression, inspect study history, reset progress, manage investigations user action with audit note
Monitoring /monitoring operations lead, on-call operator inspect tutoring latency, grounding failures, build failures, export failures, and cost anomalies alert tuning or incident escalation
Incidents /incidents incident commander unify monitoring, complaints, moderation, rights, and learner investigations in one shell runbook execution and workspace routing
Operations /operations platform operator, safety reviewer inspect transcripts, grounded build evidence, and model cost footprints investigation evidence
Audit Trail /audit support lead, compliance reviewer validate operator actions, queue history, and escalation chronology audit confirmation or escalation

3. Role Model and Decision Rights#

Role Can decide directly Must escalate when Required companion surface
Curriculum reviewer publication package, grounded course build, pedagogical prompt change rights posture is unclear, grounded evidence is incomplete, or integrity concerns exist /reviews, /rights, /operations
Rights reviewer allowed-use posture, provenance sufficiency, licensing blocker status source ownership is disputed, consent is missing, or legal interpretation is required /rights, /audit
Moderation operator generated-content takedown, tutor abuse acknowledgement, moderation escalation learner safety risk, repeated abuse pattern, or policy ambiguity exists /content, /incidents, /audit
Learner-support operator complaint acknowledgement, investigation open, complaint resolution recommendation grade dispute affects certification, learner trust risk is severe, or complaint links to live incident /complaints, /users, /audit
Learner-support lead progression pause/resume, progress reset, investigation resolution action could affect entitlements, institutional reporting, or legal/compliance review /users, /audit
Operations lead alert threshold update, monitoring lane triage, operational routing multi-lane degradation, cost anomaly with launch impact, or grounding failure is widespread /monitoring, /incidents, /operations
Incident commander runbook selection, lane ownership, downgrade/close decision rights, safety, or compliance signoff is still unresolved /incidents, /monitoring, /complaints, /content, /rights, /audit

No operator may approve a Metis artifact or close a Metis incident without leaving an auditable rationale in the relevant workspace.

4. Shift-Start and Shift-End Routine#

4.1 Shift Start#

  1. Open / and inspect quick actions, active counts, and recent operator activity.
  2. Open /monitoring and confirm whether any critical panel shows open or escalated incidents.
  3. Open /incidents and confirm whether an incident commander is already assigned for open cross-lane issues.
  4. Open /reviews, /rights, and /complaints and note any queue with breached SLA or unresolved escalations.
  5. Open /audit and confirm the previous shift left explicit rationale for any queue item still in progress.

4.2 Shift End#

  1. Ensure every changed item has an audit-backed decision, note, or escalation.
  2. Ensure open incidents have an owner, next action, and destination workspace.
  3. Ensure paused learner cases or rights blockers are visible in /audit.
  4. Record any exception-approved unresolved issue in the launch-readiness lane.
  5. Hand off unresolved high-severity work in the incident commander channel.

5. Standard Workflow Playbooks#

5.1 Review Queues#

Use /reviews when handling:

  • publication packages
  • source bundles
  • grounded course builds
  • tutor persona changes
  • pedagogy prompt changes
  • model-routing policy changes
  • avatar teacher profiles
  • voice packs
  • likeness consent records

Required sequence:

  1. Confirm the queue type and current status.
  2. Inspect the detail payload and evidence summary.
  3. If the artifact references external material, verify rights posture in /rights.
  4. If the artifact affects tutor behavior or learner safety, inspect related traces in /operations.
  5. Record an approve or reject decision with a rationale that would make sense to the next reviewer without extra context.

Approval must not proceed when grounding evidence, rights posture, or approval history is incomplete.

5.2 Source Rights#

Use /rights whenever imported or generated educational material depends on third-party source rights.

Required sequence:

  1. Confirm provenance coverage and allowed-use posture.
  2. Inspect unknown-rights counts, licensing compatibility, and linked source artifacts.
  3. If the material is already referenced from a review queue, return to /reviews only after a rights decision exists.
  4. If the item blocks publication, escalate it into /incidents.
  5. Record whether the item is approved for release, review-only, or blocked.

5.3 Moderation#

Use /content for generated educational content risk, tutor abuse reports, and academic-integrity incidents.

Required sequence:

  1. Confirm incident type, severity, and current operator owner.
  2. Inspect generated output, learner context, and any policy signals.
  3. If the issue affects an active learner trust event, jump into /complaints or /incidents.
  4. Record the moderation outcome and note the policy basis.
  5. Verify the action appears in /audit.

5.4 Learner Complaints#

Use /complaints when a learner reports content accuracy, tutor behavior, grading, or export issues.

Required sequence:

  1. Filter to the complaint lane and open the case detail.
  2. Confirm the learner, course/session context, and current status.
  3. Decide whether the case can be acknowledged and resolved locally or needs an investigation or cross-workspace escalation.
  4. If the case affects progression or history, open /users/[userId].
  5. If the case indicates broad platform degradation, open /incidents.
  6. Record the action and confirm it appears in /audit.

5.5 Learner Operations#

Use /users and /users/[userId] for operator actions that directly affect a learner account.

Required sequence:

  1. Confirm the learner identity and active investigations.
  2. Inspect study history and current course activity before pausing or resetting anything.
  3. Use pause/resume only when continuity or safety posture requires it.
  4. Use progress reset only with explicit operator rationale.
  5. Confirm every user action is visible in /audit.

5.6 Monitoring and Incident Command#

Use /monitoring to identify platform risk and /incidents to run the shared runbook across queue types.

Required monitoring sequence:

  1. Inspect critical panels for tutoring latency, grounding failures, build failures, export failures, and model cost anomalies.
  2. If a lane is locally recoverable, tune alerts or assign follow-up work.
  3. If the issue spans multiple lanes or creates customer trust risk, route it into /incidents.

Required incident-command sequence:

  1. Open the incident detail and confirm category, severity, owner, and source workspace.
  2. Work the runbook steps in order.
  3. Use the escalation routes to return to the resolving workspace instead of tracking work in chat only.
  4. Close the incident only after the resolving workspace action and audit entry both exist.

6. Training Program#

6.1 Initial Qualification#

An operator is not qualified for unsupervised Metis work until all of the following are complete:

  1. Read this runbook and the current Metis scorecard.
  2. Walk the live route map in the admin app with a trainer.
  3. Complete one shadow session for review/rights work.
  4. Complete one shadow session for complaints/user operations.
  5. Complete one shadow session for monitoring/incident command.
  6. Demonstrate audit-trail verification after every operator action.

6.2 Scenario Drills#

Every new operator must complete and pass the following drills:

Drill Required route flow Pass condition
Publication review with rights blocker /reviews -> /rights -> /audit operator blocks approval until rights posture is explicit
Tutor behavior complaint escalation /complaints -> /incidents -> /users/[userId] -> /audit complaint becomes investigation-backed with clear owner
Monitoring spike to shared incident /monitoring -> /incidents -> source workspace -> /audit incident runbook is followed and routed back into the correct lane
Moderation action with learner impact /content -> /incidents -> /audit moderation decision includes policy rationale and incident linkage
Learner progression intervention /users/[userId] -> /audit operator leaves durable rationale for pause/reset action

6.3 Ongoing Calibration#

Weekly:

  • review one closed incident and one closed complaint in /audit
  • compare approval/rejection rationales across at least two operators
  • confirm alert tuning changes still align with actual failure posture

Monthly:

  • rerun at least one cross-lane escalation drill
  • review rights and moderation decisions for policy drift
  • confirm this runbook still matches the live admin route set

7. Evidence Requirements#

An operator action is incomplete unless all applicable evidence exists:

Action type Required evidence
Review approval or rejection queue item, rationale, linked evidence summary, audit trail entry
Rights decision provenance posture, allowed-use posture, blocker state, audit trail entry
Moderation decision policy basis, affected content/session reference, audit trail entry
Complaint action learner context, operator note, resulting complaint state, audit trail entry
Learner management action user context, operator rationale, resulting status/history change, audit trail entry
Incident closure runbook completion, destination-workspace action, audit trail entry

If the evidence is missing, the operator must reopen or escalate the item rather than silently closing it.

8. Escalation Paths#

Area Primary owner Escalate to
Review queues and course publication Metis Product + Metis Admin Engineering OSHUN Platform and relevant substrate owner
Source rights and provenance Compliance + rights reviewer Legal/compliance lead
Moderation and tutor behavior Safety + moderation operator Safety lead and incident commander
Learner complaints and user operations Support operations Support lead and product owner
Monitoring and incidents Platform operations Program operations and incident commander
Audit-trail gaps or permission ambiguity Security + platform owner OSHUN Security lead

When a workflow crosses subsystem boundaries, the escalation target must follow the Metis ownership matrix rather than local convenience.

9. Currency Rule#

This runbook must be reviewed whenever any of the following changes:

  • a new Metis admin route becomes launch-critical
  • a queue type, incident type, or user action type changes
  • scorecard status for operator documentation and training changes
  • release-gate ownership or escalation policy changes

If the live app and this runbook disagree, the disagreement is a launch-readiness issue and must be resolved before the operator workflow is treated as current.