Runbook — Yemaya study workspace: rollout

Checklist: YSD-22058 (rollout) · drill kind rollout Decision owner role: the service owner for svc-study-workspace Machinery: apps/yemaya/svc-study-workspace/src/persistence/migrate.ts

The nearest thing to this that existed was a lock-timeout docblock in migrate.ts, addressed to whoever edits migrations. It is good and it is about one schema change. Nothing described putting a service build in front of traffic in stages, what signal halts it, or who decides — and having nobody to name is how a rollout continues past the first bad signal.

Running this document IS the drill. Record it in procedure-exercise-register.json.

Detection — what tells you a stage is going wrong#

Not "a customer tells us". In order of how early they fire:

1. The migration refused to apply. Every migration runs under lock_timeout and aborts transactionally and unrecorded when it queues behind a held lock. This is by design and is not a failure: the rollout retries after the blocking work clears. Treat repeated aborts as a signal that something long-running holds a lock, not as a reason to raise the timeout.
2. Readiness stops passing on the new stage (readiness.ts).
3. Error rate or latency on the new stage diverges from the old one. The comparison is against the stage still serving, not against an absolute — a deploy during a traffic spike looks bad on absolutes and fine on the diff.

The stages, and who decides#

Rollout time objective: 45 minutes, from applying the migration to either completing or halting. Set 2026-08-15, and derived rather than chosen: the four stages above are budgeted at 5 + 5 + 15 + 10 = 35, and the objective is that plus ten minutes of headroom, rounded up to the next five — the same rule yemaya-study-recovery.md and yemaya-study-rollback.md use. An objective with no headroom over the plan meant to meet it is missed by any run that hits one snag, and one without an objective at all cannot be drilled: evaluateDrill refuses a record that has no duration to be measured against.

The objective is not met by halting quickly. Halting is a legitimate outcome of stage 4 and stops the clock; going past 45 minutes still holding traffic at a stage is the miss.

Step 4 is the one this document exists for. The decision to continue is a decision, taken by a named role, on the signals in front of them. A rollout that proceeds because nobody said stop has not been decided.

Forward-only#

migrate.ts treats a checksum change on an applied migration as a hard error. There is no "edit the migration and redeploy": a fix ships as a new migration file. If you are tempted to amend one, you are about to make two databases disagree about what has run.

Communication#

Done means#

• Every instance serves the new build, readiness passes, and the three signals match the pre-rollout baseline.
• study_schema_migrations holds exactly the migrations the build expects.
• The result is recorded in the procedure exercise register, naming who ran it, the environment, and how long each stage took.

Known shortfall#

The migration half is exercised on every change by migrations-rollout.integration.spec.ts, which applies a real migration over seeded volume against live PostgreSQL with a concurrent writer and checks for stalls. The build half — staged traffic, the halt signal, the decision — is exercised by nothing. Until a drill works through it, this document describes an intended procedure rather than a rehearsed one, and that gap is the whole distinction the register is keeping.