# Runbook — Model or Workflow Rollback

> Roll back a recently promoted model version, retriever version, evaluator
> version, or Isis generation workflow that is causing regression in quality,
> safety, latency, cost, or compliance.

## Scope

Applies when a promotion via `release-taxonomy.ts` (event
`release.model.promoted` or `release.workflow.promoted`) needs to revert to the
previous certified version. Targets include:

- Assistant LLM versions per persona.
- Sophia retriever, reranker, and grounding-evaluator versions.
- Metis tutor evaluator and assessment evaluator.
- Lilith policy/refusal versions (their joint baselines with the assistant LLM).
- Isis generation workflows (image/video/audio/3D).
- Veritas claim/source/counterclaim evaluator.

This runbook handles the rollback. The persona-version rollback uses
`persona-rollback.md`. Provider-route failover uses `provider-failover.md`.

## When to roll back

A rollback fires when **any one** of:

- An evaluation benchmark in `evaluation-manifest.ts` regresses past its
  regression-blocking threshold.
- An alert from `alerts-manifest.ts` tied to the promoted version fires
  (`model-health-degraded`, `model-rollback`, `citation-flag-rate`,
  `persona-policy-violation`).
- The release-captain post-promotion review identifies a customer-visible issue.
- A T&S escalation identifies a safety regression tied to the version.
- A cost or latency budget overshoot exceeds the per-version SLO.

Rollback is the default mitigation. Stay rolled back unless a forward fix clears
the original blocker.

## Severity classification

| Sev  | Condition                                                                                                              |
| ---- | ---------------------------------------------------------------------------------------------------------------------- |
| sev1 | Safety regression — unsafe-pass rate above threshold, refusal-bypass detected, persona-policy violation in production. |
| sev1 | Cost regression with 10× run-away or budget exhaustion in <1 h.                                                        |
| sev2 | Quality regression past the evaluation threshold; customer-visible degraded answers.                                   |
| sev3 | Latency or marginal cost regression; no customer-visible defect.                                                       |

## Detection signals

- Alerts: `model-health-degraded`, `model-rollback`, `citation-flag-rate`,
  `persona-policy-violation`, `generation-pipeline-failover`,
  `grounding-low-confidence-spike`.
- Dashboards: `dashboards-assistant.ts` quality + compliance;
  `dashboards-service-health.ts` per-target panels; `dashboards-customer-kpi.ts`
  conversion and satisfaction panels.
- Audit signals: `admin.model.promoted`, `admin.model.evaluation_failed`,
  `admin.workflow.promoted` per `admin-taxonomy.ts`.

## Rollback procedure

1. **Confirm the regression**:
   - Pull the current evaluation-manifest run for the affected target.
   - Compare against the pre-promotion baseline run.
   - Confirm the regressed metric is past the regression-blocking threshold from
     `evaluation-manifest.ts`.
   - Save the comparison artifact to the audit store under the incident ID — the
     postmortem will reference it.
2. **Identify the previous certified version**: read the `release-taxonomy.ts`
   release history for the target — the prior `release.model.certified` record
   is the rollback destination. If no prior certified record exists (first
   release of the target), rollback is not possible — fall back to the
   kill-switch path (mitigation step 4).
3. **Invalidate the promoted cache**: any cached answers, embeddings, reranks,
   or generation outputs tagged with the regressed version must be invalidated
   before the rollback closes — otherwise customers continue to see the bad
   version's output. The cache-invalidation hook emits
   `release.cache.invalidated` with the version tag.
4. **Pin the previous version**: call the release-taxonomy rollback hook; the
   hook emits `release.model.rolled_back` (or `.workflow.rolled_back`) with the
   source and destination version, the operator, and the reason code. Customer
   surfaces and assistant routes pick up the rollback on their next request.
5. **Cohort verification**: sample 20 in-flight customer sessions or jobs that
   hit the rolled-back target. Verify the response carries the prior version's
   metadata (model id, retriever id, workflow id).
6. **Re-run the gating evaluation** post-rollback to confirm the metric
   recovered to baseline; the run is added to the audit store.

## Kill-switch path (when no prior version exists)

If the target is a first-time release with no prior certified version:

1. Disable the target entirely via the target's kill-switch flag in
   release-taxonomy.
2. Customer surfaces fall back to the documented degraded mode (assistant
   grounded-mode disabled, Isis generation queue paused, etc.).
3. The on-call team for the target follows the matching domain runbook to manage
   the customer-visible degraded mode.

## Communication cadence

| Phase      | Template ID                                                  | Audience    | Cadence                          |
| ---------- | ------------------------------------------------------------ | ----------- | -------------------------------- |
| Internal   | `incident.internal.slack.investigating`                      | internal    | at decision                      |
| Identified | `incident.customer.status_page.identified`                   | status_page | for customer-visible regressions |
| Mitigating | `incident.customer.status_page.mitigating`                   | status_page | every 30 min                     |
| Resolved   | `incident.customer.status_page.resolved` + `.email.resolved` | both        | within 60 min                    |
| Postmortem | `incident.customer.status_page.postmortem`                   | status_page | within 5 biz day                 |

For safety-related rollbacks (refusal bypass, unsafe-pass), the customer
notification must specifically state the safety scope and the recertified
version. T&S co-authors the copy.

## Escalation

Standard 5/5/15/30/60 escalation. For sev1 safety rollbacks, page T&S lead and
the model-ops lead at minute 0.

## Recovery verification

- Evaluation gating metrics within thresholds for the rolled-back version.
- All caches invalidated for the regressed version (no stale serves for 60 min
  as confirmed by trace-attribute sampling).
- Customer-KPI panels (`dashboards-customer-kpi.ts`) recovered to baseline for
  30 min.
- No new alerts of the same class for 30 min.
- Audit pipeline records the rollback and the recertified version.

## Post-incident

- Postmortem must include: the regression metric and its threshold, the
  promotion record that introduced it, the rollback record, customer-impact
  estimate, cost/latency overshoot if any, and the remediation owner for the
  forward fix.
- File a promotion-process improvement ticket if the regression slipped past the
  evaluation pipeline — either the threshold was too loose or the benchmark
  missed the regressed dimension.
- If the regression was safety-class, add the new failure to the safety
  benchmark fixture so the next promotion catches it.
- The forward fix follows the standard model-promotion gate; do not shortcut
  promotion to "re-fix" a previously regressed version.