# Runbook — Yemaya study workspace: rollout

**Checklist:** YSD-22058 (`rollout`) · drill kind `rollout` **Decision owner
role:** the service owner for `svc-study-workspace` **Machinery:**
`apps/yemaya/svc-study-workspace/src/persistence/migrate.ts`

The nearest thing to this that existed was a lock-timeout docblock in
`migrate.ts`, addressed to whoever edits migrations. It is good and it is about
one schema change. **Nothing described putting a service build in front of
traffic in stages, what signal halts it, or who decides** — and having nobody to
name is how a rollout continues past the first bad signal.

**Running this document IS the drill.** Record it in
`procedure-exercise-register.json`.

## Detection — what tells you a stage is going wrong

Not "a customer tells us". In order of how early they fire:

1. **The migration refused to apply.** Every migration runs under `lock_timeout`
   and aborts _transactionally and unrecorded_ when it queues behind a held
   lock. This is by design and is not a failure: the rollout retries after the
   blocking work clears. Treat repeated aborts as a signal that something
   long-running holds a lock, not as a reason to raise the timeout.
2. **Readiness stops passing** on the new stage (`readiness.ts`).
3. **Error rate or latency on the new stage diverges from the old one.** The
   comparison is against the stage still serving, not against an absolute — a
   deploy during a traffic spike looks bad on absolutes and fine on the diff.

## The stages, and who decides

| #   | Action                                                                        | Owner role                | Expected |
| --- | ----------------------------------------------------------------------------- | ------------------------- | -------- |
| 1   | Apply migrations against production, forward-only                             | service owner             | 5 min    |
| 2   | Bring one instance of the new build up, serving no traffic; confirm readiness | on-call engineer          | 5 min    |
| 3   | Move a small share of traffic to it; hold and watch the three signals above   | on-call engineer          | 15 min   |
| 4   | Complete the rollout, or halt and go to `yemaya-study-rollback.md`            | **service owner decides** | 10 min   |

**Rollout time objective: 45 minutes**, from applying the migration to either
completing or halting. Set 2026-08-15, and derived rather than chosen: the four
stages above are budgeted at 5 + 5 + 15 + 10 = 35, and the objective is that
plus ten minutes of headroom, rounded up to the next five — the same rule
`yemaya-study-recovery.md` and `yemaya-study-rollback.md` use. An objective with
no headroom over the plan meant to meet it is missed by any run that hits one
snag, and one without an objective at all cannot be drilled: `evaluateDrill`
refuses a record that has no duration to be measured against.

The objective is **not** met by halting quickly. Halting is a legitimate outcome
of stage 4 and stops the clock; going past 45 minutes still holding traffic at a
stage is the miss.

Step 4 is the one this document exists for. **The decision to continue is a
decision, taken by a named role, on the signals in front of them.** A rollout
that proceeds because nobody said stop has not been decided.

## Forward-only

`migrate.ts` treats a checksum change on an applied migration as a hard error.
There is no "edit the migration and redeploy": a fix ships as a new migration
file. If you are tempted to amend one, you are about to make two databases
disagree about what has run.

## Communication

| Audience                                                      | Within      |
| ------------------------------------------------------------- | ----------- |
| The on-call channel — rollout started, and each stage entered | immediately |
| Affected tenants, if a stage is halted                        | 30 min      |

## Done means

- Every instance serves the new build, readiness passes, and the three signals
  match the pre-rollout baseline.
- `study_schema_migrations` holds exactly the migrations the build expects.
- The result is recorded in the procedure exercise register, naming who ran it,
  the environment, and how long each stage took.

## Known shortfall

The migration half is exercised on every change by
`migrations-rollout.integration.spec.ts`, which applies a real migration over
seeded volume against live PostgreSQL with a concurrent writer and checks for
stalls. **The build half — staged traffic, the halt signal, the decision — is
exercised by nothing.** Until a drill works through it, this document describes
an intended procedure rather than a rehearsed one, and that gap is the whole
distinction the register is keeping.