Disciplines · Runbooks

Runbook: V1 Game-Day Plan

one is a sentence about the command rather than a reading from the estate.**

11sections14 minread
Markdown source
On this page

Owner: production operations duty officer (rota:prod-ops). Last reviewed: 2026-08-14. Version: 1.

The ten scenarios in WORKBENCH_GAME_DAY_SCENARIOS say what disasters this estate can have. The nine v1-dr-*.md runbooks say what an operator does during one. This document says how the estate is deliberately broken on a Tuesday afternoon so that those two can be tested before an incident tests them: the data the drill runs against, the controls that inject each fault, the reading that establishes the fault landed, what watches, what is timed, where the drill is not allowed to go, who stops it, what is kept, and what would have shown the drill failed.

It covers §S10.12.d. The plans are published as a machine-checkable statement at the end and graded by libs/oshun/workbench-kit/tools/plan-scan.mjs.

0. The rule this document is written under#

A fault control is a claim that something broke, and every field describing one is a sentence about the command rather than a reading from the estate.

The contract shipped by §S10.12.a can see that a control has a reversal and that the reversal runs on abort. It cannot see whether the injection does anything. Three readings taken on the development box on 2026-08-14 are why that gap matters, and all three are reproducible with node scripts/operations/v1-fault-control.mjs --self-test:

  1. docker exec <container> kill -9 1 exits 0 and kills nothing. The kernel discards SIGKILL sent to a PID namespace's init from inside that namespace, so the process the drill meant to crash keeps serving. kill -TERM 1 does work, because postgres installs a handler. The strongest signal is the one that does nothing, and it is the one an operator reaches for. A drill reading the exit code of its own injection would record a fault, measure a perfectly healthy estate, and report a recovery time for an outage that never began.

  2. The obvious controls do not produce the same fault. With a five-second client connect timeout:

    control what the client saw elapsed
    docker stop could not translate host name 539 ms
    docker network disconnect could not translate host name 359 ms
    docker pause timeout expired 5407 ms
    iptables -j DROP timeout expired 5370 ms
    iptables -j REJECT Connection refused 316 ms

    Those are three faults, not three spellings of one. A client that fails at name resolution never opens a socket and gives up in 400 ms; a client whose packets are dropped waits out its own connect timeout. A host that is really gone keeps its name — the record is in DNS, a config file or an environment variable, and the packets go nowhere — so docker stop rehearses a graceful deregistration and measures a recovery time with every caller's connect timeout removed from it. The contract now refuses a control whose measured presentation is not one the scenario's impairment can produce.

  3. A trap is a promise a process makes about its own death. §S10.12.a added EXIT/INT/TERM traps to the two mutating drills, which is strictly better than the linear restore they had. It is still not a reversal for the case that needs one: a drill paused a container, was killed with SIGKILL, and the container was still paused afterwards with nothing left to unpause it. SIGKILL, an OOM kill and a reboot are exactly the deaths a trap cannot honour — and reversesOnAbort: true is satisfied by a trap. So v1-fault-control.mjs writes the reversal record to disk before it injects, and --reverse-all drains that ledger from a process with no relationship to the one that applied it.

So: for every fault control, name the observation that distinguishes the injected estate from a healthy one, and confirm the injection by taking it.

1. Dataset#

production-like, and never production.

The estate is one Hetzner box running infra/hetzner/docker-compose.yml. There is no second environment, so a drill runs either against the live estate or against a copy stood up from the same compose file and the most recent off-box backup (infra/hetzner/backup/offbox-backup.sh). Every plan below declares production-like and every fault control declares affectsRealActors: false, because the drills below are run against that copy.

This is a real constraint and not a preference. The development box shares one postgres, one redis and one minio between several worktrees, and the destructive controls would take their data with them. That is why the safety boundary in §5 is enforced by the driver rather than written down here.

2. Fault controls#

Six, all of them applied, confirmed and reversed on this box before being written here. node scripts/operations/v1-fault-control.mjs --list prints the registry; --probe reports which are available on the host you are standing on.

id applies presents as destroys data
fc:freeze-process docker pause timeout no
fc:drop-inbound iptables -A INPUT … -j DROP timeout no
fc:refuse-inbound iptables -A INPUT … -j REJECT refused no
fc:deregister-container docker stop name-unresolvable no
fc:flush-keyspace redis-cli FLUSHALL absent-data yes
fc:corrupt-heap-page zero one 8 KiB heap page, server down wrong-answer yes

fc:deregister-container is registered so that a plan naming it is refused. It is the control an operator reaches for first, its presentation is name-unresolvable, and the impairment table does not accept that for an unavailable scenario. Leaving it out would have left the most-reached-for control undescribed rather than described and disallowed.

2.1 What the corruption control measured, and why it changes the grading#

Zeroing page 1 under a 2000-row table left select count(*) returning 1815 with no error, max(id) still 2000, and the connection healthy. 185 rows gone, every liveness check passing, and the query that lost them reporting success.

Corrected on 2026-08-14 (S10.12.g, retest:D-F4) — this section used to say the wrong thing, and it said it confidently. It read: "It is silent because data_checksums is off, which is how the running estate is configured … the one instrument that would have turned this into a loud error is disabled." The remedy that follows from that sentence was measured, four arms, same table, same injection:

page data data_checksums what a count(*) returned
random off ERROR: invalid page in block 1
random on the same ERROR, plus the checksum
zeros off 1815 of 2000 rows, no error
zeros on 1815 of 2000 rows, no error

Checksums were never the instrument that would have made this loud. An all-zero page is a valid unused page to PostgreSQL by design, so a store that lost one answers cheerfully with fewer rows however the cluster was initialised; a garbled page was already refused by the page-header check before any checksum was involved. POSTGRES_INITDB_ARGS: --data-checksums is now set in infra/hetzner/docker-compose.yml — it buys the diagnosis on the first two rows (which arm, and by how much), costs nothing at initdb, and applies only to clusters initialised after that change, which means every clean-host restore and not the running production volume.

The grading consequence is unchanged and is the part that was right: no drill on this estate can be graded by "did the store stay up", because the silent case stays silent. Detection of a zeroed page has to come from a row count compared against something outside the store — the reconciler, scripts/operations/v1-restore-reconcile.mjs. That is finding F-4 in §10, still open, with its statement corrected rather than its verdict flipped.

3. Observers#

This section records the 2026-08-14 exercise baseline. At that time, infra/hetzner/docker-compose.yml declared no Prometheus, Grafana, or external alert route, so the exercise correctly retained F-1 and F-2. The current source now defines a profile-gated private Prometheus/Alertmanager/Grafana plane, answer-capable probes and exporters, plus two external receivers. The distinct dead-man receiver is the observer outside the host failure domain. See v1-observability-game-day.md.

That source change is not retroactive evidence and is not yet a passed live retest. Until the profile is deployed on the exact candidate and fresh external incident/dead-man receipts pass admission, D-F1 and D-F2 remain open. The historical plan below continues to describe what the 2026-08-14 artifacts actually exercised; Task 13.7 supplies the production-bound successor rather than silently relabelling the old run.

What actually observed the estate during that exercise was 14 Docker healthchecks, and every one of them ran against localhost inside the container it checked:

text
postgres   pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}
redis      redis-cli ping
minio      curl -f http://localhost:9000/minio/health/live
bff        curl -fsS http://localhost:8080/health

A paused container cannot run its own healthcheck; a box loss takes all fourteen. So no historical plan below uses a container healthcheck as an observer — the contract refuses one whose every route is inside the blast radius, and following that refusal is what produced the two observers that were used:

  • obs:fault-control-probe — v1-fault-control.mjs --confirm, run from the operator's workstation, which is not on the estate. It classifies what a client sees into refused / timeout / name-unresolvable / healthy, and it detects unavailable.

    It requires an answer, not a handshake, and that is a measurement rather than a preference: the first version settled healthy on the socket's connect event, and against a container frozen with docker pause it reported healthy — the kernel completes the three-way handshake from the listen backlog whether or not any process will ever read from it. A frozen dependency accepts every connection and answers none. The instrument written to catch a blind observer was blind to the fault that hides best.

  • obs:replay-inventory-census — scripts/operations/v1-replay-inventory.mjs --output before the drill and --compare BEFORE AFTER after it. It counts records per store key and per state, so it detects corrupt: it is the only observer on this estate that can tell 1815 rows from 2000.

A corrupt dependency still answers, so an observer reading it is not blinded — the contract was repaired to say so, because as shipped it refused the only observer a corruption drill can have.

4. Timers#

All six instants in TIMED_INSTANTS, and each has something that stamps it:

instant stamped by
fault-injected recordedAtMs in .game-day/fault-control-ledger.json, written before the injection
detected the first --confirm reading that differs from healthy
declared the duty officer, in the drill log
restore-started infra/hetzner/backup/box-loss-restore-drill.sh start
restore-finished the same script's report
verified scripts/operations/v1-replay-inventory.mjs --compare returning reconciled

detected is the one the estate could not previously produce: §S10.11 recorded that all ten rollback rehearsal records carry observed-RTO and no detection instant, and DrillTiming.detectedAtMs had no producer. --confirm is that producer. Recovery time runs from injection, because a game day knows when the fault started — it caused it.

5. Safety boundaries#

Enforced by the driver, not by this paragraph:

  1. A destructive control refuses any container not carrying oshun.game-day.disposable=true. The driver puts that label only on containers it created itself. Pointing fc:flush-keyspace at the shared development redis exits non-zero with the reason, before anything is flushed.
  2. The reversal is recorded before the fault is applied. The worst case is a ledger entry for an injection that never happened, and reversing something already healthy is a no-op. The other order leaves a fault nothing knows about, in exactly the window a drill is most likely to be stopped.
  3. --self-test exits 3 when it cannot run. A run that could not happen is not a run that passed. That is §S10.12.a's finding 2 — all(.status != "fail") reading a skip as a success — and repeating it in the driver written to end it would be the same defect.
  4. --reverse-all exits non-zero for anything still applied, including the destructive controls that have no automatic reversal. Their ledger entry names the restore procedure somebody has to run.

6. Abort#

Called by the production operations duty officer (rota:prod-ops) — a named rota, not a collective; the contract refuses a team noun because in the moment somebody has to actually say it.

Calling it runs v1-fault-control.mjs --reverse-all, which drains the ledger rather than relying on the drill process still being alive to honour a trap. Abort conditions are per plan below and all of them share one: any reading that a real actor's request failed, since the drills run against a copy and no real request should reach them at all.

7. Evidence capture#

Kept for every drill, so the report can be re-read by somebody who was not in the room:

  • .game-day/fault-control-ledger.json — what was applied, and when
  • the --apply and --confirm JSON for every control, carrying the reading that confirmed the injection and its elapsed time
  • the pre-drill and post-drill v1-replay-inventory.mjs outputs and the --compare result
  • the box-loss-restore-drill.sh report, including nextRequiredControl and safeToResumeAutonomy
  • the six stamped instants

8. Success criteria#

Every criterion names the observation that would have refuted it, because 'Detect partition within 5m' is a sentence whose truth is decided afterwards by whoever writes the report, and "the detection timestamp minus the injection timestamp exceeds 300000 ms" is a comparison two people cannot disagree about.

The recovery-point comparison is owed only by a drill that can destroy data. A postgres crash replays its log and loses nothing — measured: the row committed immediately before docker kill --signal=KILL was there after the restart — and a network denial loses nothing by construction. Demanding a recovery-point criterion from those drills is demanding one that passes whatever the estate does. So the two plans whose controls destroy data carry one, and the three whose controls do not, do not.

9. The ten scenarios#

Five have plans. Five are declared unrehearsable on this estate, and the declaration is checked in both directions: planFaults refuses a plan that declares itself unrehearsable and ships a control anyway, and the scan refuses a declaration whose control the estate has since grown. Declaring is not the same as omitting — an omitted plan is scenario-with-no-plan, which reads as an oversight rather than as a measured impossibility.

scenario plan control
gd:regional-loss rehearsed fc:freeze-process, every service
gd:database-loss rehearsed fc:freeze-process on postgres
gd:blob-store-loss rehearsed fc:freeze-process on minio
gd:queue-loss rehearsed fc:freeze-process on redis
gd:relational-corruption rehearsed fc:corrupt-heap-page
gd:signing-service-outage unrehearsable no signing service is deployed
gd:signing-key-compromise unrehearsable same, and rotation is not injection
gd:model-provider-outage unrehearsable needs the ISIS admin flag API
gd:search-index-loss unrehearsable the index has no process of its own
gd:bridge-host-loss unrehearsable the enrolled host is not ours

Each declaration cites the estate:

  • gd:signing-service-outage / gd:signing-key-compromise. infra/hetzner/docker-compose.yml declares no signing service; a grep for signing across it and the development compose returns nothing. There is no process to interrupt and no key to rotate, so an injection would be a rehearsal against a dependency this estate does not run.
  • gd:model-provider-outage. The provider is outside the estate and the only control the repository has for it is scripts/isis/run_provider_outage_drill.sh, which injects by PATCHing feature flags against a live ISIS admin API. That API is not deployed here, so the control cannot be exercised — which is also why §S10.12.e is recorded as not locally actionable.
  • gd:search-index-loss. DEPENDENCY_SUBSTRATE puts dep:search-index on dep:primary-store, and the estate compose declares no Elasticsearch or other search process. The index is the relational store, so every injection that takes it away is gd:database-loss under another name.
  • gd:bridge-host-loss. The enrolled host is a workstation the estate does not own and cannot reach. The recovery is on the far side of a boundary we do not control, which is also why its recovery time is the longest in the register.

9.1 One target that cannot be injected, in a plan that otherwise can#

In the 2026-08-14 baseline, gd:regional-loss named eight targets and dep:telemetry-pipeline had no service on the deployed estate (§3). The plan was still rehearsable — freezing every application service was a real drill of a real box loss — but one named dependency could not be interrupted. That is why the historical scan reports F-1. Current source supplies the target, but a Task 13.7 live run must prove the local alert path and the external dead-man path before the finding can close. dep:search-index is in the same position for the opposite reason: freezing postgres takes it, because it is postgres.

10. Findings, for §S10.12.g#

id severity finding
F-1 high At exercise time the estate deployed no telemetry pipeline, so every plan's default observer reported over something that did not exist.
F-2 high At exercise time all 14 healthchecks ran inside the checked container, leaving the estate's only deployed observability inside every blast radius.
F-3 medium The §S10.12.a register derives the regional blast radius from the development compose, not from infra/hetzner/docker-compose.yml.
F-4 high A zeroed page is silent: 185 of 2000 rows vanished from count(*) with no error, and data_checksums does not change that (§2.1).
F-5 medium A trap-based reversal does not survive SIGKILL; a killed drill leaves the fault applied with nothing to reverse it.
F-6 low Five of the ten registered scenarios cannot be rehearsed on this estate, four of them because the dependency is not deployed.

F-1/F-2 remediation now exists in source, but their follow-ups stay open until the source is deployed and the supervised external-delivery/dead-man retest is admitted. F-5's production-bound successor likewise uses durable armed reversal state and second-process recovery; it is not closed by code inspection alone.

json
{
  "id": "game-day-plan:v1",
  "version": "1",
  "driver": "scripts/operations/v1-fault-control.mjs",
  "plans": [
    {
      "id": "plan:gd:regional-loss",
      "scenarioId": "gd:regional-loss",
      "dataset": "production-like",
      "faultControls": [
        {
          "id": "fc:freeze-process",
          "targets": [
            "dep:primary-store",
            "dep:blob-store",
            "dep:job-queue",
            "dep:search-index",
            "dep:read-cache",
            "dep:telemetry-pipeline",
            "dep:identity",
            "dep:notification-channel"
          ],
          "applies": "docker pause every service in infra/hetzner/docker-compose.yml",
          "reverses": "docker unpause the same services, or v1-fault-control.mjs --reverse-all",
          "reversesOnAbort": true,
          "affectsRealActors": false,
          "presentsAs": "timeout",
          "confirmedBy": "v1-fault-control.mjs --confirm reports timeout against postgres, redis, minio and the bff from off the estate",
          "movesTheRecoveryPoint": false
        }
      ],
      "observers": [
        {
          "id": "obs:fault-control-probe",
          "watches": "what a client sees from the operator workstation, off the estate",
          "observesThrough": [
            "out-of-band: the operator workstation, off the estate"
          ],
          "detects": ["unavailable"]
        }
      ],
      "timedInstants": [
        "fault-injected",
        "detected",
        "declared",
        "restore-started",
        "restore-finished",
        "verified"
      ],
      "abort": {
        "statement": "abort on any reading that a request from a real actor failed, or if the box does not answer 10 minutes after the reversal",
        "calledBy": "production operations duty officer (rota:prod-ops)",
        "reversesFaultControls": true
      },
      "evidence": [
        ".game-day/fault-control-ledger.json",
        "the --apply and --confirm JSON for every service",
        "the six stamped instants",
        "the box-loss-restore-drill.sh report"
      ],
      "successCriteria": [
        {
          "id": "sc:regional-rto",
          "statement": "sign in and authorize any request at all are back inside their 15-minute promise",
          "refutedBy": "verified minus fault-injected exceeds 900000ms",
          "comparedAgainst": "recovery-time-objective"
        },
        {
          "id": "sc:regional-detected",
          "statement": "the outage is detected from off the estate rather than from a report",
          "refutedBy": "no --confirm reading differs from healthy before the operator declares",
          "comparedAgainst": "observation"
        }
      ],
      "announcedToActors": true,
      "unrehearsableBecause": null
    },
    {
      "id": "plan:gd:database-loss",
      "scenarioId": "gd:database-loss",
      "dataset": "production-like",
      "faultControls": [
        {
          "id": "fc:freeze-process",
          "targets": ["dep:primary-store"],
          "applies": "docker pause postgres",
          "reverses": "docker unpause postgres, or v1-fault-control.mjs --reverse-all",
          "reversesOnAbort": true,
          "affectsRealActors": false,
          "presentsAs": "timeout",
          "confirmedBy": "v1-fault-control.mjs --confirm fc:freeze-process reports timeout after the connection is accepted and never answered",
          "movesTheRecoveryPoint": false
        }
      ],
      "observers": [
        {
          "id": "obs:fault-control-probe",
          "watches": "what a client sees from the operator workstation, off the estate",
          "observesThrough": [
            "out-of-band: the operator workstation, off the estate"
          ],
          "detects": ["unavailable"]
        }
      ],
      "timedInstants": [
        "fault-injected",
        "detected",
        "declared",
        "restore-started",
        "restore-finished",
        "verified"
      ],
      "abort": {
        "statement": "abort on any reading that a request from a real actor failed",
        "calledBy": "production operations duty officer (rota:prod-ops)",
        "reversesFaultControls": true
      },
      "evidence": [
        ".game-day/fault-control-ledger.json",
        "the --apply and --confirm JSON",
        "the six stamped instants"
      ],
      "successCriteria": [
        {
          "id": "sc:database-rto",
          "statement": "open an item is back inside its 60-minute promise",
          "refutedBy": "verified minus fault-injected exceeds 3600000ms",
          "comparedAgainst": "recovery-time-objective"
        }
      ],
      "announcedToActors": true,
      "unrehearsableBecause": null
    },
    {
      "id": "plan:gd:blob-store-loss",
      "scenarioId": "gd:blob-store-loss",
      "dataset": "production-like",
      "faultControls": [
        {
          "id": "fc:freeze-process",
          "targets": ["dep:blob-store"],
          "applies": "docker pause minio",
          "reverses": "docker unpause minio, or v1-fault-control.mjs --reverse-all",
          "reversesOnAbort": true,
          "affectsRealActors": false,
          "presentsAs": "timeout",
          "confirmedBy": "v1-fault-control.mjs --confirm fc:freeze-process reports timeout where the health endpoint answered in 4ms before",
          "movesTheRecoveryPoint": false
        }
      ],
      "observers": [
        {
          "id": "obs:fault-control-probe",
          "watches": "what a client sees from the operator workstation, off the estate",
          "observesThrough": [
            "out-of-band: the operator workstation, off the estate"
          ],
          "detects": ["unavailable"]
        }
      ],
      "timedInstants": [
        "fault-injected",
        "detected",
        "declared",
        "restore-started",
        "restore-finished",
        "verified"
      ],
      "abort": {
        "statement": "abort on any reading that a request from a real actor failed",
        "calledBy": "production operations duty officer (rota:prod-ops)",
        "reversesFaultControls": true
      },
      "evidence": [
        ".game-day/fault-control-ledger.json",
        "the --apply and --confirm JSON",
        "the six stamped instants"
      ],
      "successCriteria": [
        {
          "id": "sc:blob-rto",
          "statement": "upload an asset is back inside its 120-minute promise",
          "refutedBy": "verified minus fault-injected exceeds 7200000ms",
          "comparedAgainst": "recovery-time-objective"
        }
      ],
      "announcedToActors": true,
      "unrehearsableBecause": null
    },
    {
      "id": "plan:gd:queue-loss",
      "scenarioId": "gd:queue-loss",
      "dataset": "production-like",
      "faultControls": [
        {
          "id": "fc:freeze-process",
          "targets": ["dep:job-queue"],
          "applies": "docker pause redis",
          "reverses": "docker unpause redis, or v1-fault-control.mjs --reverse-all",
          "reversesOnAbort": true,
          "affectsRealActors": false,
          "presentsAs": "timeout",
          "confirmedBy": "v1-fault-control.mjs --confirm fc:freeze-process reports timeout where redis answered an inline command in 5ms before",
          "movesTheRecoveryPoint": false
        }
      ],
      "observers": [
        {
          "id": "obs:fault-control-probe",
          "watches": "what a client sees from the operator workstation, off the estate",
          "observesThrough": [
            "out-of-band: the operator workstation, off the estate"
          ],
          "detects": ["unavailable"]
        }
      ],
      "timedInstants": [
        "fault-injected",
        "detected",
        "declared",
        "restore-started",
        "restore-finished",
        "verified"
      ],
      "abort": {
        "statement": "abort on any reading that a request from a real actor failed",
        "calledBy": "production operations duty officer (rota:prod-ops)",
        "reversesFaultControls": true
      },
      "evidence": [
        ".game-day/fault-control-ledger.json",
        "the --apply and --confirm JSON",
        "the six stamped instants",
        "the pre-drill and post-drill v1-replay-inventory.mjs outputs"
      ],
      "successCriteria": [
        {
          "id": "sc:queue-rto",
          "statement": "move an item through a stage is back inside its 60-minute promise",
          "refutedBy": "verified minus fault-injected exceeds 3600000ms",
          "comparedAgainst": "recovery-time-objective"
        }
      ],
      "announcedToActors": true,
      "unrehearsableBecause": null
    },
    {
      "id": "plan:gd:relational-corruption",
      "scenarioId": "gd:relational-corruption",
      "dataset": "production-like",
      "faultControls": [
        {
          "id": "fc:corrupt-heap-page",
          "targets": ["dep:primary-store"],
          "applies": "zero one 8KiB heap page of the target relation while the server is stopped",
          "reverses": "restore the relation from the recovery point with box-loss-restore-drill.sh; the fault has no rollback",
          "reversesOnAbort": true,
          "affectsRealActors": false,
          "presentsAs": "wrong-answer",
          "confirmedBy": "the census comparison falls from 2000 to 1815 while the store keeps answering, measured by --self-test",
          "movesTheRecoveryPoint": true
        }
      ],
      "observers": [
        {
          "id": "obs:replay-inventory-census",
          "watches": "the per-store record counts against the pre-drill inventory",
          "observesThrough": ["dep:primary-store"],
          "detects": ["corrupt"]
        }
      ],
      "timedInstants": [
        "fault-injected",
        "detected",
        "declared",
        "restore-started",
        "restore-finished",
        "verified"
      ],
      "abort": {
        "statement": "abort if the census cannot be taken, since a corruption drill with no pre-drill count cannot be graded",
        "calledBy": "production operations duty officer (rota:prod-ops)",
        "reversesFaultControls": true
      },
      "evidence": [
        ".game-day/fault-control-ledger.json",
        "the pre-drill and post-drill v1-replay-inventory.mjs outputs and the --compare result",
        "the six stamped instants",
        "the box-loss-restore-drill.sh report"
      ],
      "successCriteria": [
        {
          "id": "sc:corruption-rpo",
          "statement": "no committed row is lost beyond the capture skew the estate can actually achieve",
          "refutedBy": "the restored census differs from the pre-drill census by more rows than the capture skew accounts for",
          "comparedAgainst": "recovery-point-objective"
        },
        {
          "id": "sc:corruption-rto",
          "statement": "open an item is back inside its 60-minute promise",
          "refutedBy": "verified minus fault-injected exceeds 3600000ms",
          "comparedAgainst": "recovery-time-objective"
        },
        {
          "id": "sc:corruption-detected",
          "statement": "the corruption is detected by a census rather than by an actor reporting a missing item",
          "refutedBy": "no census reading differs from the pre-drill count before the operator declares",
          "comparedAgainst": "observation"
        }
      ],
      "announcedToActors": true,
      "unrehearsableBecause": null
    },
    {
      "id": "plan:gd:signing-service-outage",
      "scenarioId": "gd:signing-service-outage",
      "dataset": "production-like",
      "faultControls": [],
      "observers": [],
      "timedInstants": [],
      "abort": null,
      "evidence": [],
      "successCriteria": [],
      "announcedToActors": false,
      "unrehearsableBecause": "infra/hetzner/docker-compose.yml declares no signing service, so there is no process on this estate to interrupt"
    },
    {
      "id": "plan:gd:signing-key-compromise",
      "scenarioId": "gd:signing-key-compromise",
      "dataset": "production-like",
      "faultControls": [],
      "observers": [],
      "timedInstants": [],
      "abort": null,
      "evidence": [],
      "successCriteria": [],
      "announcedToActors": false,
      "unrehearsableBecause": "no signing service is deployed and no key exists to disclose; rotation is a recovery procedure rather than an injection, so there is nothing to apply or reverse"
    },
    {
      "id": "plan:gd:model-provider-outage",
      "scenarioId": "gd:model-provider-outage",
      "dataset": "production-like",
      "faultControls": [],
      "observers": [],
      "timedInstants": [],
      "abort": null,
      "evidence": [],
      "successCriteria": [],
      "announcedToActors": false,
      "unrehearsableBecause": "the provider is outside the estate and the repository's only control for it, scripts/isis/run_provider_outage_drill.sh, injects by PATCHing feature flags against a live ISIS admin API that is not deployed here"
    },
    {
      "id": "plan:gd:search-index-loss",
      "scenarioId": "gd:search-index-loss",
      "dataset": "production-like",
      "faultControls": [],
      "observers": [],
      "timedInstants": [],
      "abort": null,
      "evidence": [],
      "successCriteria": [],
      "announcedToActors": false,
      "unrehearsableBecause": "DEPENDENCY_SUBSTRATE puts dep:search-index on dep:primary-store and the estate compose declares no search process, so the index is the relational store and every injection that takes it away is gd:database-loss under another name"
    },
    {
      "id": "plan:gd:bridge-host-loss",
      "scenarioId": "gd:bridge-host-loss",
      "dataset": "production-like",
      "faultControls": [],
      "observers": [],
      "timedInstants": [],
      "abort": null,
      "evidence": [],
      "successCriteria": [],
      "announcedToActors": false,
      "unrehearsableBecause": "the enrolled bridge host is a workstation the estate does not own and cannot reach, so no control here can interrupt it"
    }
  ]
}