Disciplines · Runbooks

Runbook — Yemaya study workspace: model withdrawal and replacement

When a vendor withdraws a model, every claim that cites it stops being reproducible.

6sections3 minread
Markdown source
On this page

Checklist: YSD-22058 (model replacement) · drill kind model-replacement · YSD-17106 Decision owner role: the analysis owner for the study workspace Machinery: libs/yemaya/study-workspace/src/continuity/disaster-recovery.ts

⚠️ This is one of the two scenarios this workspace classes as irrecoverable (IRRECOVERABLE_SCENARIOS, with key compromise). A plan claiming clean recovery from it is refused by the plan validator, and permanentLosses is a required field for exactly this reason.

What existed was the analysis, not the procedure: disaster-recovery.ts states what a withdrawal destroys. Nothing said who marks the affected claims, how they are found, or what is told to the people relying on them.

What is actually lost#

When a vendor withdraws a model, every claim that cites it stops being reproducible. The claim is still there and its provenance is still complete; it can never be re-derived or independently re-verified again, because the thing that produced it no longer exists.

Re-running the analysis on a replacement model does not restore the old claim. It produces a new claim, from a different model, which may say something else. Treating the new output as a repair of the old one silently rewrites what a learner was told and why. The old claim is marked, not overwritten.

The steps#

# Action Owner role Expected
1 Establish the withdrawal date and whether the vendor offers a frozen version analysis owner 60 min
2 Compute the affected set — every output whose provenance cites the withdrawn model analysis owner see below
3 Mark those claims as no longer reproducible, preserving their text and provenance analysis owner 120 min
4 Tell the people relying on them analysis owner within 72 h
5 Select a replacement, and run it as a NEW analysis leaving the marked claims in place analysis owner —

Marking objective: 240 minutes, from learning of the withdrawal to every affected claim being marked as no longer reproducible — steps 1 and 3, budgeted at 60 + 120 = 180, plus ten minutes of headroom rounded up to the next five, then rounded to the hour because a four-hour piece of work is not scheduled in five-minute increments. Set 2026-08-15.

There is deliberately no objective covering step 5. Selecting a replacement model is a judgement about quality, and the register's own verification asks that the replacement was compared against the previous model on the same inputs — which takes as long as the comparison takes. An objective that covered it would be a deadline on how quickly somebody may decide what is good enough, and that deadline is the reason replacement models get adopted without comparison.

Step 4 has its own deadline (72 hours, in Communication below) and it is not part of this objective: telling people is a promise about how long they may go untold, and a procedure that met it while leaving every claim unmarked has met nothing.

Step 2 is the one that may not be possible#

Analysis outputs record a pipelineVersion string. Whether that string identifies the withdrawn model is the question this whole procedure turns on, and it is not something this document can promise. Before anything else:

  • Confirm pipelineVersion on the affected outputs resolves to a specific model and version, not merely to a pipeline name.
  • If it does not, the affected set cannot be computed, and that is the finding — the workspace cannot say which of its claims depended on the withdrawn model. Record it as such rather than approximating by date range. An approximated set marks claims that were fine and misses claims that were not.

Communication#

Audience Within
The analysis owner and the service owner immediately
Learners whose notebooks or reports cite a marked claim 72 hours

Done means#

  • Every affected claim carries a mark saying it is no longer reproducible, and its original text and provenance are intact.
  • The people relying on them have been told, in those terms — not "an update to our models", which conceals the only thing that matters.
  • permanentLosses on the continuity plan for this scenario names what is gone.
  • Recorded in the procedure exercise register.

Known shortfall#

Nothing exercises this — zero mechanism runs. No suite drives a withdrawal, so unlike the store-outage and expiry procedures, no part of this has been observed working. It is an intended procedure written from the module's own account of the scenario, and the first drill should expect to find step 2 harder than it reads.