- Owner: UNASSIGNED — no named person or rota owns this register. Each finding below has an owner; the register itself does not.
- Last reviewed: 2026-08-14
- Machine-readable:
docs/releases/v1/verification/game-day/2026-08-14/follow-up-register.json - Graded by
libs/oshun/workbench-kit/tools/follow-up-scan.mjs(pnpm run verify:operations:v1-follow-ups)
Every finding S10.12.a through S10.12.g measured, with a severity, an owner, a deadline and a backlog reference — and, for the ones that are closed, the run that closed them. This document and the JSON are graded against each other: the scan fails if a state stated here differs from the register, so a finding cannot be closed by editing prose.
1. What an entry has to carry, and why each field is refused when empty#
followUpFaults in @oshun/workbench-kit/game-day refuses a finding with no
severity ("nothing orders this against the others, so it is worked on last"), no
owner, an owner that is a collective ("a collective does not close a finding"),
no deadline ("so it is closed by the next drill finding it again"), no backlog
link ("the finding lives only in a drill report, which nobody reads on an
ordinary Tuesday"), and — the one this cell turns on — a critical finding
closed without a retest.
The scan adds what the contract cannot see:
- The owner is read, not assigned. Every owner is a contact in the shipped dependency register, and where an entry names a dependency, one of that dependency's owners. The register cannot invent a name to satisfy the field.
- The deadline is arithmetic:
raisedAtMs + policy(severity), with the policy stated once — critical 14 days, major 30, minor 90. A row quietly given six more weeks fails. - Closure is backed by a run: a closed entry citing a retest must match a
record in
follow-up-retests.jsonwhose verdict isfixedand whose finishing instant is the register'sretestedAtMs. - Nothing measured is missing: every finding the acceptance matrix emitted appears here, under its own id or as an alias of the entry that absorbed it.
- This document is graded against the JSON cell by cell — state, severity,
owner, deadline, and the tally sentence under the table. A reader acts on the
severity and the date as much as on the word
closed, so a row that agrees on one column and lies on another is not a smaller defect. Twenty-seven negative controls mutate the real register and the real document; the scan exits 2 rather than reporting at all if any of them goes undetected.
The limit of the owner field, stated. The rota identifiers above are real — they are what the S10.5 dependency register records — and
docs/operations/on-call.mdnames no person and holds no rotation config, so none of them resolves to somebody carrying a pager. The owner column says which rota is accountable, not who will answer. That gap isA-11's neighbour and is stated in every scenario runbook's §1.And nobody has agreed to the deadline policy.
agreedByisnullin the register. The windows are the ones this cell proposes; they are enforced by arithmetic so that changing them is a commit rather than a drift.
2. The register#
| id | severity | state | owner | due | finding |
|---|---|---|---|---|---|
A-1 |
critical | closed | production operations duty officer | 2026-08-28 | the documented game-day invocation reported a pass having injected nothing |
A-2 |
critical | closed | production operations duty officer | 2026-08-28 | all(.[]; .status != "fail") reads a skip as a pass |
A-3 |
major | closed | production operations duty officer | 2026-09-13 | DRY_RUN defaults to true in all three drills |
A-4 |
critical | closed | production operations duty officer | 2026-08-28 | jq -r '.isReadOnly // true' returns true for every input |
A-5 |
major | closed | production operations duty officer | 2026-09-13 | nothing scheduled any drill, so there was no rotation |
A-6 |
minor | open | workbench-platform on-call | 2026-11-12 | the scenario library's targets are not dependency ids |
A-7 |
minor | open | workbench-platform on-call | 2026-11-12 | four of the ten classes have no scenario in that library |
A-8 |
minor | open | workbench-platform on-call | 2026-11-12 | its objectives are prose graded by numbers a facilitator types in |
A-9 |
minor | open | workbench-platform on-call | 2026-11-12 | the whole library has no caller |
A-10 |
critical | closed | production operations duty officer | 2026-08-28 | neither mutating drill had a trap |
A-11 |
major | open | production operations duty officer | 2026-09-13 | ten rollback records carry an RTO and no detection instant |
A-13 |
major | open | production operations duty officer | 2026-09-13 | RPO/RTO are declared per substrate rather than per capability |
B-1 |
major | closed | production operations duty officer | 2026-09-13 | no runbook mentioned the V1 recovery path |
B-2 |
major | closed | production operations duty officer | 2026-09-13 | keyRecoveryVerified: true was a literal with no check behind it |
B-3a |
minor | open | production operations duty officer | 2026-11-12 | content-service and computer-use are verified by directory existence |
C-1 |
critical | open | workbench-platform on-call | 2026-08-28 | the reconciler every runbook names has no producer for its input |
C-2 |
minor | open | workbench-platform on-call | 2026-11-12 | V1 has no outbox; the replay document declares the absence |
C-3 |
major | open | delivery pipeline on-call | 2026-09-13 | the lease fence is real and its reclaim loop is deployed nowhere |
D-F1 |
major | open | observability on-call | 2026-09-13 | monitoring now exists in source; exact-candidate live alert delivery is not yet admitted |
D-F2 |
major | open | observability on-call | 2026-09-13 | external dead-man exists in source; missing-signal delivery is not yet admitted |
D-F3 |
minor | open | workbench-platform on-call | 2026-11-12 | the blast-radius comment cites the development compose |
D-F4 |
major | open | workbench-platform on-call | 2026-09-13 | a zeroed page vanishes silently, and checksums do not change that |
D-F5 |
minor | open | production operations duty officer | 2026-11-12 | a trap-based reversal does not survive SIGKILL |
D-F6 |
minor | open | production operations duty officer | 2026-11-12 | five of ten scenario classes cannot be injected on this estate |
F-f1 |
critical | open | workbench-platform on-call | 2026-08-28 | a returning box writes into the key space of the estate that replaced it |
F-f2 |
critical | closed | delivery pipeline on-call | 2026-08-28 | the backup verifies a redis checkpoint the restore never reads |
F-f4 |
major | open | workbench-platform on-call | 2026-09-13 | no degraded mode exists during a regional loss |
F-f5 |
minor | open | delivery pipeline on-call | 2026-11-12 | work written after the capture is lost and still sits on the returned box |
F-f6 |
minor | open | observability on-call | 2026-11-12 | a connect-only probe is blind to a frozen dependency |
G-1 |
major | closed | delivery pipeline on-call | 2026-09-13 | a failed INFO keyspace inside a pipeline recorded a fabricated zero |
Seven criticals, twelve majors, eleven minors. Ten closed.
3. The retests#
scripts/operations/v1-follow-up-retest.mjs runs each of these against real
containers and writes
docs/releases/v1/verification/game-day/2026-08-14/follow-up-retests.json.
Every retest carries an arm about the state before the repair, because a
retest that only shows the repaired code passing cannot tell a fix from a check
that was always going to pass. For three of the four that arm is the code as
it was, run against the same input. For retest:key-recovery it cannot be:
the pre-fix behaviour was the absence of a check, and running nothing against
a bad input accepts it by definition, so that arm measures the artifact instead
— the drill no longer writing the field as a literal, and calling the verifier
that produces it. An arm asserting a remembered outcome would be a sentence
dressed as a measurement, and there are none.
retest:drill-verdict → fixed (closes A-1, A-2, A-3, A-4, A-10)#
The invocation docs/reference/ci-cd.md gives verbatim, run again. The old
predicate all(.[]; .status != "fail"), applied to the checks that run
produced, still returns true — one passing check and one skipped is the
strongest pass it can express. The script now returns
overall.outcome=inconclusive, overall.pass=false, naming both reasons, and the
twelve tests that shipped with the repair are green.
retest:F-f2 → fixed (closes F-f2, and B-3 for redis and objects)#
Twenty-five keys, a forced BGSAVE, and two restores. Handed only the
checkpoint the backup verifies, an --appendonly yes redis comes up with 0 of
25 keys and answers PONG — the pre-fix check accepts it. The repaired check
refuses it by name. Handed the whole archived data directory, the same check
accepts 25 of 25.
retest:key-recovery → fixed (closes B-2)#
The required set is derived from every ${VAR:?} in the deployed compose. A
complete recovery environment is accepted; the same file with
MINIO_ROOT_PASSWORD emptied is refused by name. The third arm reads
box-loss-restore-drill.sh and measures that the literal
keyRecoveryVerified:true is gone and the verifier is called — because the
pre-fix code had no check at all to re-run, the field being a constant in a jq
template.
retest:D-F4 → remedy-does-not-fix-it (closes nothing)#
The retest that refuted its own finding. D-F4 was raised as "data_checksums
is off, so page corruption is silent", and the four arms say otherwise:
| page data | data_checksums |
what count(*) returned |
|---|---|---|
| random | off | ERROR: invalid page in block 1 |
| random | on | the same ERROR, plus the checksum |
| zeros | off | 1815 of 2000 rows, no error |
| zeros | on | 1815 of 2000 rows, no error |
An all-zero page is a valid unused page to PostgreSQL. The remedy adds a
diagnosis to a garbled page — which was already refused — and changes nothing
about the fault that was reported. POSTGRES_INITDB_ARGS: --data-checksums is
set anyway, because the diagnosis is worth having and costs nothing at initdb;
the finding stays open with its statement corrected, and detection of a
zeroed page has to come from a row count compared against something outside the
store.
4. Re-running it#
pnpm run run:operations:v1-follow-up-retest # the four retests, ~4 minutes
pnpm run verify:operations:v1-follow-ups # grades the register, 27 negative controls
Both run in the follow-ups job of
.github/workflows/v1-disaster-recovery.yml, retests first — the grading
refuses a closure whose retestedAtMs is not the instant a run finished, so
grading against a stale retest report would be grading against a claim.
The retest harness exits 3 when docker is unreachable and 1 when a retest cannot decide. A retest that refutes its finding exits 0 — that is a result, and the record is what matters.
The scan fails in two ways and they mean different things. A malformed
register is a defect in this document or the JSON: a name that resolves to
nobody, a deadline that does not follow the policy, a closure with no run behind
it. An overdue register is well-formed and the work is late — which is what a
deadline is for, and why it exits non-zero rather than printing a note. The
output separates them. Set OSHUN_FOLLOW_UP_NOW_MS to see the overdue path
before a deadline arrives; at 2027-01-01 all twenty open findings report late.
5. What this register does not establish#
- It does not establish that any open finding is being worked on. It records who is accountable and by when.
- It does not establish that the deadlines are achievable. Nobody has agreed to them; the policy is enforced arithmetically so that changing one is visible.
- It does not cover findings from outside S10.12. The rollback rehearsal records, the on-call documentation and the reconciler's missing producer are referenced here because game days found them, not because this register owns them.