Runbook: V1 Disaster Recovery (box loss) (gd:regional-loss)

Owner: production operations duty officer (rota:prod-ops). Last reviewed: 2026-09-14. Version: 3.

The V1 stack runs on a single host. This runbook covers losing it: the box is unreachable, destroyed, or has to be abandoned, and the estate has to be rebuilt somewhere else from the off-box backup.

Every section below is written from scripts that exist in this repository, and each step cites the file it runs. §8 is the one that cannot be written from the repository, because a fence is not something a script can supply. It is marked ABSENT rather than filled in, so that missingRunbookSections in @oshun/workbench-kit/game-day reports it and nobody mistakes an unwritten section for a satisfied one.

Related: database-backups.md covers the portable pg_dump path for a cluster that is still reachable, and object-storage-backups.md the object store. Neither covers this: the V1 off-box path is restic to a remote repository, and the restore is not finished when the restore finishes (§10). The nine narrower scenarios in the same register have their own runbooks — signing-service outage, relational corruption, signing-key compromise, queue loss, model-provider outage, search-index loss, blob-store loss, database loss and bridge-host loss. This one is the union of most of them, which is what makes it the regional scenario.

1. Detection and declaration#

Detection signal. The off-box backup writes a receipt whose age is bounded by OSHUN_OFFBOX_HEALTH_MAX_AGE_SECONDS (infra/hetzner/backup/offbox-backup-healthcheck.sh). A receipt that stops being refreshed is the first machine-observable sign that the box is gone.

The source now supplies two complementary paths. The profile-gated Prometheus plane in infra/hetzner/docker-compose.yml detects service/process loss while the host remains alive. A distinct external dead-man receiver must page when its once-per-minute OshunDeadMansSwitch notification stops, so whole-host or collector loss is distinguishable from health. See v1-observability-game-day.md. The local plane is still inside the regional blast radius; the receiver is deliberately not. Until Task 13.7 admits a live dead-man receipt for the exact deployed candidate, the backup receipt remains a corroborating detector and the external path is a source-level control, not claimed operational evidence.

• Detection authority: production operations duty officer, contact rota:prod-ops, as the S10.5 dependency register records for every dependency this scenario takes away.
• Declaration authority: production operations duty officer, same contact. The two are the same role here and that is a property of the estate rather than an oversight: there is one operations rota.
• Gap, stated, and it is narrower and worse than "there is no on-call". docs/operations/on-call.md exists and describes a weekly primary/secondary rotation with an escalation ladder. It names no person, and it holds no rotation config — while instructing the reader to "open a PR against the rotation config in this doc", which is not in it. Its own Owner: is platform team, a collective. None of the register's four rota: identifiers appears in it or in docs/operations/alert-routing.md, so the name above cannot be resolved to whoever is actually carrying the pager.
• A local evaluator cannot announce its own disappearance. The Hetzner Prometheus and Alertmanager remain co-located with the services, so their ordinary incident route ends with the host. The separately provisioned dead-man receiver is responsible for detecting that silence from outside the platform domain. Production deploy validates that the incident and dead-man endpoints are distinct external HTTPS receivers; a supervised missing-signal test and fresh receipt are still required before relying on it operationally.
• Whoever declares must be reachable over something that does not run on the V1 host — the host is the thing that is gone. See §2.
• Record the detection instant. DETECTED_AT_EPOCH is a real input to scripts/isis/run_runpod_disaster_recovery_gameday.sh and the field DrillTiming.detectedAtMs in @oshun/workbench-kit/restore-consistency exists for it. A recovery time measured from the moment somebody ran the restore command excludes detection and decision, which is most of a real recovery.

2. Communications#

Nothing in the repository sends these; they are the operator's to send, and they are listed with deadlines so that a missed one is a fact rather than an impression.

The channel must not be the estate, and the register now says so. dep:notification-channel is a target of gd:regional-loss: the sender is the BFF (apps/oshun/bff/src/auth/verification-email-sender.ts), which is on the lost host. Messages routed over it during a regional loss do not arrive, and runbookFaults refuses a runbook that tries.

Gap, stated. The estate has no out-of-band channel implemented. The three rows above travel over whatever the operator can reach from outside the box, and this repository does not provide it. That is the same class of gap as §8: a human control standing in for a mechanism.

3. Dependency assumptions#

The recovery assumes exactly these, and nothing else:

• The restic repository is reachable and is not on the V1 host. infra/hetzner/backup/offbox-backup.sh refuses a local repository outright: RESTIC_REPOSITORY must not be local to the V1 host.
• Docker is available on the recovery host (docker command is unavailable is a hard failure in box-loss-restore-drill.sh).
• find, jq, sha256sum and stat are on PATH — the drill checks each.

It assumes nothing about the lost host. If a step needs something only the lost host had, that step is wrong.

4. Recovery infrastructure#

A clean host with no running Oshun stack. Not a dependency in the S10.5 register and never will be: it is provisioned for the recovery.

box-loss-restore-drill.sh enforces both halves of "clean":

docker ps --format '{{.Names}}' | grep -Eq '^oshun-(staging|production)(-|$)'
  # -> "an Oshun runtime is already running; this is not a clean-host drill"
find "${TARGET_DIR}" -mindepth 1 -maxdepth 1 -print -quit
  # -> "target directory is not empty"

OSHUN_CLEAN_HOST_ID is required and is recorded in the proof as targetHostId, so the report says which host the estate came back on.

5. Secrets#

Two, and both must be obtainable without the estate:

• RESTIC_REPOSITORY and the restic password (RESTIC_PASSWORD or RESTIC_PASSWORD_FILE), supplied in the recovery env file. The drill fails with restic recovery password is absent if neither is set.
• The recovery env file itself must deny group and other access; the drill refuses it otherwise (recovery env permissions must deny group/other access).

The drill's comment states the rule the rest of this runbook depends on: the recovery env "is operator-controlled and must not be fetched from the repository being recovered".

Closed on 2026-08-14 (S10.12.g, finding B-2). This section used to read: "the restore proof asserts keyRecoveryVerified: true and there is no step in the script that exercises a key beyond the restic password itself." There is now: infra/hetzner/backup/verify-restored-state.sh keys checks the restored stack.env against every ${VAR:?} in infra/hetzner/docker-compose.yml — the set the stack refuses to start without — and the drill writes keyRecoveryVerified as the value that check produced instead of as a literal. The required set is derived from the compose rather than listed, so a new required secret cannot be forgotten here. Retested both ways (retest:key-recovery): a complete environment is accepted, and one missing MINIO_ROOT_PASSWORD is refused by name.

6. Restore order#

Run infra/hetzner/backup/box-loss-restore-drill.sh:

infra/hetzner/backup/box-loss-restore-drill.sh \
  production /path/to/recovery.env /path/to/empty-target /path/to/proof.json

It performs, in order: restic check --read-data-subset → restic restore latest --tag oshun-v1 --tag <stack> → manifest schema check (oshun.v1-offbox-backup-manifest.v1) → per-dump SHA-256 verification → PostgreSQL into a scratch container → Redis → MinIO mirror → stack.env presence.

That order is safe only because nothing is serving traffic. The drill runs into an isolated docker network with no route in. When the estate is brought back for actors, the order is the other way round for the stores that reference each other:

This is the reverse of the CAPTURE order in @oshun/workbench-kit/restore-consistency, and deliberately so: capture the referring store first so inserts leave orphans, restore the referenced store first so no row is exposed pointing at bytes that are not back yet. restoreOrderFaults in @oshun/workbench-kit/game-day checks a proposed order against the same reference graph.

The full order, and it is longer than the three stores above. A regional loss takes ten registered dependencies, not four: docker/docker-compose.yml declares the whole estate in one file with no placement constraint, so the telemetry, the cache, the session authority and the notification sender go with the data stores. Bring them back in this order:

1. dep:identity — nothing else is reachable by an actor until sessions are, and nothing in the list below is verifiable by anybody who cannot sign in.
2. dep:telemetry-pipeline — before the stores, not after. It is the instrument the rest of this restore is watched through, and a restore observed only by the operator's terminal is one whose failures are noticed in the order somebody happens to look.
3. dep:notification-channel — the sender, so §2's messages stop needing the out-of-band route the moment they can stop.
4. dep:blob-store — before the rows that reference it.
5. dep:primary-store — the rows.
6. dep:read-cache — after its source, and empty. A cache restored from a snapshot serves the estate's state as of the snapshot, which is the one failure in this list that looks like success.
7. dep:primary-store-scan — correct as soon as the rows are.
8. dep:search-index — start Qdrant empty and reproject personalization vectors from PostgreSQL; it is last because no Qdrant snapshot is authoritative.
9. dep:job-queue, then 10. dep:job-worker-pool — the pool rests on the queue, and a pool started first reports itself healthy while doing nothing.

dep:signing-service is not in the numbered list because a regional loss does not take it: there is no signing service in the compose file, the signer is in-process (see v1-dr-signing-service-outage.md). If it is ever restored alongside these, it goes before dep:primary-store, as the third row of the table above says.

7. Validation#

The drill's own floors, which are real checks and not assertions:

• every *.dump has a .sha256 beside it and the bytes hash to it;
• checksum_count > 0 — a snapshot containing no PostgreSQL dumps fails;
• database_count == checksum_count — every dump has metadata and every metadata has a dump;
• each restored database answers SELECT count(*) >= 0 FROM pg_catalog.pg_class;
• Redis holds what the capture recorded. The manifest carries inventory.redisKeyCount and inventory.redisPersistentKeyCount, taken from INFO keyspace at the instant of the forced BGSAVE, and verify-restored-state.sh redis refuses a restore that answers PING with an empty keyspace, one short of the persistent keys (keys with no TTL cannot expire, so a shortfall is loss), or one holding more than was captured (which would mean this is not the snapshot that was restored). A snapshot whose manifest predates those counts is refused rather than accepted on the PING.
• MinIO holds what was mirrored: inventory.minioObjectCount against the objects on the restored volume.
• stack.env carries every value the stack refuses to start without (§5).

Closed on 2026-08-14 (S10.12.g, finding F-f2). This section used to read: "Redis, MinIO, the file state and the configuration are asserted true in the report from steps that an EMPTY store also passes." S10.12.f measured what that cost: redis-server --appendonly yes — how this estate runs it — creates a fresh AOF on start and never reads dump.rdb, so a host handed the checkpoint the backup forces and verifies comes up holding 0 of 25 entries and answers PONG. The counts above are the repair, and retest:F-f2 runs both arms: the pre-fix check accepts that empty restore, the repaired check refuses it by name, and the whole archived data directory is accepted at 25 of 25.

Still open (B-3a): the content-service and computer-use directories are checked for existence and nothing else. An empty one passes.

8. Fencing — ABSENT for a promoted primary#

Correction to an earlier version of this runbook, which said there is no fencing anywhere in this estate. That was a claim about a grep — for split-brain, fence, fencing and STONITH across docs/, infra/ and scripts/ — and the estate's one real fencing token is in none of those directories and uses none of those words.

It is live_media_pipeline_job_attempts.lease_token, in libs/shared/live-media/src/sql-media-pipeline-job-store.ts. Every heartbeat, lease expiry and completion carries AND lease_token = $4 AND status = 'leased' in its UPDATE ... WHERE, so a worker whose lease was reassigned updates zero rows instead of overwriting the worker that replaced it. That is textbook lease fencing, it works, and it is the pattern the rest of this section is missing rather than a pattern the estate has never used.

What is absent is a fence for a promoted primary, which is what this scenario needs: nothing rejects the writes of a host that comes back after its replacement was promoted. A job attempt is protected from a stale worker; a store is not protected from a stale host.

Consequence, stated so that it is a decision rather than an accident: if the lost host returns to life while the recovery host is serving, there are two primaries and nothing rejects the writes of the older one. Until a fencing token covers the stores the way lease_token covers a job attempt, the recovery host must not be brought up for actors while the original host might still be reachable. That is a human control, and it is the weakest step in this runbook.

9. Failover — promoting the recovery host#

Failover here is not a switch; it is the decision to let actors reach the recovery host, and it is the step §8 makes dangerous.

1. Confirm the reconciler (§10) has run and returned no blockers. A host that is serving before reconciliation can repeat a completed provider effect.
2. Confirm the original host is unreachable and cannot come back on its own. With no fencing token, this is a human judgement and it is the one that causes split-brain when it is wrong.
3. Point DNS or the load balancer at OSHUN_CLEAN_HOST_ID. The recovery host is identified in the proof as targetHostId, so the record of where the estate went is the same artifact as the record that it came back.
4. Record the instant. Recovery time runs from the fault, not from this step.

infra/hetzner/backup/box-loss-restore-drill.sh deliberately does none of this: it restores into an isolated docker network and refuses to run at all if an Oshun runtime is present. Promotion is outside the drill and always has been.

10. Reconciliation — the restore is not finished when the restore finishes#

The proof says so itself:

{
  "status": "restore-pass-reconciliation-required",
  "safeToResumeAutonomy": false,
  "nextRequiredControl": "scripts/operations/v1-restore-reconcile.mjs"
}

Run scripts/operations/v1-restore-reconcile.mjs before resuming any autonomous or effectful work. It compares restored provider effects, publications, queues, invoices, artifacts, placements and approvals against what the outside world observed, and returns blockers.

Before starting the HTTP-serving BFF, start PostgreSQL, Redis, and an empty Qdrant, then run the bundled production reprojection entry point:

docker compose -p oshun-production \
  --env-file /opt/oshun/production/.env \
  -f /opt/oshun/bundle/docker-compose.yml \
  run --rm bff node dist/reproject-personalization-vector-index.js

The command creates or validates the collection and its userId, schemaVersion, and embeddingVersion payload indexes, removes stale points, projects every non-tombstoned PostgreSQL vector, and fails unless readback count, payload version, and dimensions match. Starting BFF repeats the same idempotent reconciliation before constructing HTTP. Production boot also requires Redis, the deletion-attestation signer, and the independent remote recovery-deletion journal; it replays every verified receipt before HTTP exists. A missing dependency, failed reprojection, failed receipt import, or failed replay keeps the BFF down. Do not route traffic around this admission boundary.

Give it expected counts. Every collection it is handed can be empty, and empty reconciles clean — an empty restore produces the same verdict as a complete one unless the expectation is supplied. Record what the estate held at the recovery point, and compare.

And the counts have to come from somewhere. Nothing in this repository writes that reconciler's input file; scripts/operations/v1-replay-inventory.mjs reads the nine state classes out of the database so there is an inventory to compare against, and refuses to report a count for a store it could not read. The state-specific procedures for folding each class back in — which work may be re-driven, which must never be, and how the operator tells them apart — are in v1-reconciliation-and-replay.md. That document is the rest of this step.

11. Failback#

Returning to a rebuilt original host is the same procedure with the roles swapped, plus one step this runbook is explicit about: take a fresh backup of the recovered estate before failing back. The recovery host has been serving; the snapshot it was built from has not been current since §6, and until a new recovery point exists the estate's most recent backup predates the disaster.

12. What this runbook does not cover#

• Partial restores (a single table, a single bucket prefix). See database-backups.md.
• A cluster that is still reachable — use the pg_dump path there; it is faster and does not need a clean host.
• Search and provenance bytes are not in the off-box backup set. The personalization Qdrant collection comes back only through the mandatory PostgreSQL reprojection above; other search recovery has its own runbook (v1-dr-search-index-loss.md) and is likewise a reprojection, which is why derived indexes are not in the backup.
• How any of this is rehearsed. This runbook is what an operator does during an incident; v1-game-day-plan.md is how the estate is deliberately broken on a Tuesday so the steps here are tested before an incident tests them. It carries the fault controls, the reading that confirms each injection landed, the observers, the timed instants, the safety boundaries and the abort — and it records which five of the ten registered scenarios cannot be rehearsed on this estate at all, with the reason for each.
• Whether the steps here actually work. v1-acceptance-matrix.md is the result of running them: seven aspects — split-brain prevention, degraded operation, queued work, user communication, monitoring and alerts, the backup of the recovered primary, and the failback in §11 above — each answered by a step of one executed drill on a disposable estate. Three of the seven currently fail, and the matrix says which and by how much. Read it before trusting §8, §9 or §11.
• What is being done about what those drills found. v1-follow-up-register.md carries every finding the game-day programme measured — thirty of them — with a severity, an owner, a deadline and, for the ten that are closed, the run that closed them. Two of the closures repaired this runbook's own restore path: §5 (keyRecoveryVerified was a literal) and §7 (a PING accepted an empty queue). The twenty still open are what an operator should expect to go wrong, ranked, with a date against each.

13. Machine-checkable statement#

The block below is the input runbookFaults in @oshun/workbench-kit/game-day grades, and libs/oshun/workbench-kit/tools/runbook-scan.mjs runs it against the scenario register. It is here rather than in a test file so that the statement the checker reads and the statement the operator reads are one object.

promotesAStandby is true and fencing is null, so the contract reports promotion-with-no-fencing against this runbook. That refusal is correct and expected: it is §8, stated in a form a machine can count.

{
  "id": "runbook:v1-regional-loss",
  "scenarioId": "gd:regional-loss",
  "version": "3",
  "detectionAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["out-of-band: a route that does not run on the V1 host"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["out-of-band: a route that does not run on the V1 host"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 300000
    },
    {
      "audience": "provider",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 3600000
    }
  ],
  "assumes": [
    "the off-box restic repository, which offbox-backup.sh refuses to place on the V1 host",
    "docker on the recovery host",
    "find, jq, sha256sum and stat on PATH"
  ],
  "recoveryInfrastructureDependencyId": "a clean host with no running Oshun stack",
  "secrets": [
    {
      "id": "RESTIC_REPOSITORY and the restic password",
      "heldInDependencyId": "the operator-controlled recovery env file",
      "recoverableOutOfBand": true
    },
    {
      "id": "every other key on the lost host (keyRecoveryVerified is unproven, see section 5)",
      "heldInDependencyId": "the operator-controlled recovery env file",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": [
    "dep:identity",
    "dep:telemetry-pipeline",
    "dep:notification-channel",
    "dep:blob-store",
    "dep:primary-store",
    "dep:read-cache",
    "dep:primary-store-scan",
    "dep:search-index",
    "dep:job-queue",
    "dep:job-worker-pool"
  ],
  "fencing": null,
  "promotesAStandby": true,
  "validation": [
    "every *.dump has a .sha256 beside it and the bytes hash to it",
    "checksum_count > 0: a snapshot containing no PostgreSQL dumps fails",
    "database_count == checksum_count",
    "each restored database answers SELECT count(*) >= 0 FROM pg_catalog.pg_class",
    "the personalization Qdrant projection command reports matching authoritative, projected and verified counts",
    "BFF boot completes independent recovery-deletion receipt import and replay before HTTP exists",
    "v1-restore-reconcile.mjs returns no blockers against recorded expected counts"
  ],
  "failback": "the same procedure with the roles swapped, plus a fresh backup of the recovered estate taken before failing back: until a new recovery point exists the estate's most recent backup predates the disaster"
}