Model operators run the model-and-workflow intake, evaluation, promotion, kill-switch, and rollback pipeline. They own the gate that sits between a candidate model or generation workflow and production traffic.
Audience and prerequisites#
- Audience: platform-engineering model-ops staff plus Sophia, Iris, Lilith, and Isis subsystem leads who run their own model/workflow promotions.
- Prerequisites: platform-engineering onboarding, RBAC current, ADR-0006 audit scope, evaluation-pipeline reading list.
- Refresh cadence: 6 months; per-promotion checklist mandatory.
Learning objectives#
By certification, a model operator can independently:
- Run the intake pipeline for a new model or workflow version (ingestion, metadata capture, provenance binding).
- Execute the evaluation pipeline (
evaluation-manifest.ts) and interpret per-scope results against regression thresholds. - Promote a version through the canary stages with cohort analysis.
- Trigger a rollback per
docs/runbooks/model-workflow-rollback.md, including cache invalidation, joint-baseline reconciliation, and audit capture. - Trip a kill switch for a target that has no certified prior version, and coordinate the customer-visible degraded mode.
- Author the promotion record: scope, evaluation artifacts, cost envelope, rollout plan, rollback plan, ownership.
Curriculum modules#
| # | Module | Duration | Format | Assessment |
|---|---|---|---|---|
| 1 | Intake pipeline and provenance binding | 90 min | hands-on with intake fixtures | 1 intake completed end-to-end |
| 2 | Evaluation pipeline scopes and thresholds | 180 min | seminar + hands-on | 4-scope evaluation interpretation test |
| 3 | Canary cohort analysis | 120 min | hands-on with canary tooling | 3 canary readouts (one promote, one hold, one roll back) |
| 4 | Promotion gate and joint baseline rule | 90 min | scenario rehearsal | 2 promotion drills (assistant LLM + Sophia retriever) |
| 5 | Rollback procedure (joint with runbook owner) | 90 min | scenario rehearsal + audit drill | rollback runbook walkthrough pass |
| 6 | Kill-switch and customer-visible degraded modes | 60 min | walkthrough of all targets | per-target degraded-mode mapping pass |
| 7 | Cost and capacity guardrails | 60 min | hands-on with cost dashboards | budget overshoot detection drill |
| 8 | Adversarial and safety evaluation | 120 min | joint with T&S | 3 safety-regression sample interpretation pass |
Promotion-record protocol#
Every promotion lands a structured record with:
- Scope: target (assistant LLM, Sophia retriever, Lilith policy, Isis workflow, evaluator, etc.); cohort (tenants, locales, traffic class); joint baseline mates if any.
- Evaluation artifacts: per-scope run IDs and outcomes from
evaluation-manifest.ts; pre-promotion vs. baseline. - Cost envelope: per-call cost, projected daily envelope, alerting threshold for budget overshoot.
- Rollout plan: canary stages (1%, 5%, 25%, 50%, 100%) and monitoring windows.
- Rollback plan: prior certified version pointer, cache invalidation scope, joint-baseline reconciliation, customer-visible degraded mode if rollback fires.
- Ownership: model operator on record, T&S co-signer for safety-class changes, compliance co-signer for regulator-scope changes.
Scenario rehearsal#
- Promote an assistant LLM under a joint baseline: certified persona + Lilith policy + new LLM version. Operator verifies the three move together; observes canary; promotes or rolls back.
- Hold a promotion on a regression: candidate fails the unsafe-pass threshold by a small margin. Operator holds; files the regression report; coordinates with model author for the forward fix.
- Roll back a Sophia retriever: citation flag rate elevated post- promotion. Operator executes the runbook; verifies cache invalidation; recertifies the recovered version.
- Kill-switch a first-time release: a brand-new evaluator has no prior version; the operator kills the target and routes customer surfaces to the documented degraded mode.
- Cost run-away: per-call cost on a recently-promoted model exceeds projected envelope by 10×. Operator triggers cost-class rollback within the documented SLO.
Certification criteria#
- All 8 modules complete with passing assessment.
- All 5 scenarios passed.
- Operator has co-led 3 promotions (one assistant, one Sophia, one Isis workflow) with a senior model operator.
- Platform-engineering lead, T&S lead (for safety-class), and finance lead (for cost guardrails) sign-off.
Tabletop drills#
- Per-promotion: pre-promotion checklist with reviewer co-sign.
- Quarterly: rollback drill against the runbook owner.
- Semiannually: cost run-away tabletop with finance and reliability.
Owner#
The platform-engineering model-ops lead owns this training. Subsystem leads (Sophia, Iris, Lilith, Isis) co-own the modules touching their targets. Updates require platform-engineering lead and T&S lead sign-off.