# Training — Model Operators

> Model operators run the model-and-workflow intake, evaluation, promotion,
> kill-switch, and rollback pipeline. They own the gate that sits between a
> candidate model or generation workflow and production traffic.

## Audience and prerequisites

- Audience: platform-engineering model-ops staff plus Sophia, Iris, Lilith, and
  Isis subsystem leads who run their own model/workflow promotions.
- Prerequisites: platform-engineering onboarding, RBAC current, ADR-0006 audit
  scope, evaluation-pipeline reading list.
- Refresh cadence: 6 months; per-promotion checklist mandatory.

## Learning objectives

By certification, a model operator can independently:

1. Run the intake pipeline for a new model or workflow version (ingestion,
   metadata capture, provenance binding).
2. Execute the evaluation pipeline (`evaluation-manifest.ts`) and interpret
   per-scope results against regression thresholds.
3. Promote a version through the canary stages with cohort analysis.
4. Trigger a rollback per `docs/runbooks/model-workflow-rollback.md`, including
   cache invalidation, joint-baseline reconciliation, and audit capture.
5. Trip a kill switch for a target that has no certified prior version, and
   coordinate the customer-visible degraded mode.
6. Author the promotion record: scope, evaluation artifacts, cost envelope,
   rollout plan, rollback plan, ownership.

## Curriculum modules

| #   | Module                                          | Duration | Format                           | Assessment                                               |
| --- | ----------------------------------------------- | -------- | -------------------------------- | -------------------------------------------------------- |
| 1   | Intake pipeline and provenance binding          | 90 min   | hands-on with intake fixtures    | 1 intake completed end-to-end                            |
| 2   | Evaluation pipeline scopes and thresholds       | 180 min  | seminar + hands-on               | 4-scope evaluation interpretation test                   |
| 3   | Canary cohort analysis                          | 120 min  | hands-on with canary tooling     | 3 canary readouts (one promote, one hold, one roll back) |
| 4   | Promotion gate and joint baseline rule          | 90 min   | scenario rehearsal               | 2 promotion drills (assistant LLM + Sophia retriever)    |
| 5   | Rollback procedure (joint with runbook owner)   | 90 min   | scenario rehearsal + audit drill | rollback runbook walkthrough pass                        |
| 6   | Kill-switch and customer-visible degraded modes | 60 min   | walkthrough of all targets       | per-target degraded-mode mapping pass                    |
| 7   | Cost and capacity guardrails                    | 60 min   | hands-on with cost dashboards    | budget overshoot detection drill                         |
| 8   | Adversarial and safety evaluation               | 120 min  | joint with T&S                   | 3 safety-regression sample interpretation pass           |

## Promotion-record protocol

Every promotion lands a structured record with:

- **Scope**: target (assistant LLM, Sophia retriever, Lilith policy, Isis
  workflow, evaluator, etc.); cohort (tenants, locales, traffic class); joint
  baseline mates if any.
- **Evaluation artifacts**: per-scope run IDs and outcomes from
  `evaluation-manifest.ts`; pre-promotion vs. baseline.
- **Cost envelope**: per-call cost, projected daily envelope, alerting threshold
  for budget overshoot.
- **Rollout plan**: canary stages (1%, 5%, 25%, 50%, 100%) and monitoring
  windows.
- **Rollback plan**: prior certified version pointer, cache invalidation scope,
  joint-baseline reconciliation, customer-visible degraded mode if rollback
  fires.
- **Ownership**: model operator on record, T&S co-signer for safety-class
  changes, compliance co-signer for regulator-scope changes.

## Scenario rehearsal

1. **Promote an assistant LLM under a joint baseline**: certified persona +
   Lilith policy + new LLM version. Operator verifies the three move together;
   observes canary; promotes or rolls back.
2. **Hold a promotion on a regression**: candidate fails the unsafe-pass
   threshold by a small margin. Operator holds; files the regression report;
   coordinates with model author for the forward fix.
3. **Roll back a Sophia retriever**: citation flag rate elevated post-
   promotion. Operator executes the runbook; verifies cache invalidation;
   recertifies the recovered version.
4. **Kill-switch a first-time release**: a brand-new evaluator has no prior
   version; the operator kills the target and routes customer surfaces to the
   documented degraded mode.
5. **Cost run-away**: per-call cost on a recently-promoted model exceeds
   projected envelope by 10×. Operator triggers cost-class rollback within the
   documented SLO.

## Certification criteria

- All 8 modules complete with passing assessment.
- All 5 scenarios passed.
- Operator has co-led 3 promotions (one assistant, one Sophia, one Isis
  workflow) with a senior model operator.
- Platform-engineering lead, T&S lead (for safety-class), and finance lead (for
  cost guardrails) sign-off.

## Tabletop drills

- Per-promotion: pre-promotion checklist with reviewer co-sign.
- Quarterly: rollback drill against the runbook owner.
- Semiannually: cost run-away tabletop with finance and reliability.

## Owner

The platform-engineering model-ops lead owns this training. Subsystem leads
(Sophia, Iris, Lilith, Isis) co-own the modules touching their targets. Updates
require platform-engineering lead and T&S lead sign-off.