# Runbook: V1 Disaster Recovery (box loss) (`gd:regional-loss`)

Owner: production operations duty officer (`rota:prod-ops`). Last reviewed:
2026-09-14. Version: 3.

> The V1 stack runs on a single host. This runbook covers losing it: the box is
> unreachable, destroyed, or has to be abandoned, and the estate has to be
> rebuilt somewhere else from the off-box backup.

Every section below is written from scripts that exist in this repository, and
each step cites the file it runs. §8 is the one that cannot be written from the
repository, because a fence is not something a script can supply. It is marked
ABSENT rather than filled in, so that `missingRunbookSections` in
`@oshun/workbench-kit/game-day` reports it and nobody mistakes an unwritten
section for a satisfied one.

Related: [database-backups.md](database-backups.md) covers the portable
`pg_dump` path for a cluster that is still reachable, and
[object-storage-backups.md](object-storage-backups.md) the object store. Neither
covers this: the V1 off-box path is restic to a remote repository, and the
restore is not finished when the restore finishes (§10). The nine narrower
scenarios in the same register have their own runbooks —
[signing-service outage](v1-dr-signing-service-outage.md),
[relational corruption](v1-dr-relational-corruption.md),
[signing-key compromise](v1-dr-signing-key-compromise.md),
[queue loss](v1-dr-queue-loss.md),
[model-provider outage](v1-dr-model-provider-outage.md),
[search-index loss](v1-dr-search-index-loss.md),
[blob-store loss](v1-dr-blob-store-loss.md),
[database loss](v1-dr-database-loss.md) and
[bridge-host loss](v1-dr-bridge-host-loss.md). This one is the union of most of
them, which is what makes it the regional scenario.

## 1. Detection and declaration

**Detection signal.** The off-box backup writes a receipt whose age is bounded
by `OSHUN_OFFBOX_HEALTH_MAX_AGE_SECONDS`
(`infra/hetzner/backup/offbox-backup-healthcheck.sh`). A receipt that stops
being refreshed is the first machine-observable sign that the box is gone.

The source now supplies two complementary paths. The profile-gated Prometheus
plane in `infra/hetzner/docker-compose.yml` detects service/process loss while
the host remains alive. A distinct external dead-man receiver must page when its
once-per-minute `OshunDeadMansSwitch` notification stops, so whole-host or
collector loss is distinguishable from health. See
[`v1-observability-game-day.md`](v1-observability-game-day.md). The local plane
is still inside the regional blast radius; the receiver is deliberately not.
Until Task 13.7 admits a live dead-man receipt for the exact deployed candidate,
the backup receipt remains a corroborating detector and the external path is a
source-level control, not claimed operational evidence.

- Detection authority: **production operations duty officer**, contact
  `rota:prod-ops`, as the S10.5 dependency register records for every dependency
  this scenario takes away.
- Declaration authority: **production operations duty officer**, same contact.
  The two are the same role here and that is a property of the estate rather
  than an oversight: there is one operations rota.
- **Gap, stated, and it is narrower and worse than "there is no on-call".**
  `docs/operations/on-call.md` exists and describes a weekly primary/secondary
  rotation with an escalation ladder. It names **no person**, and it holds no
  rotation config — while instructing the reader to "open a PR against the
  rotation config in this doc", which is not in it. Its own `Owner:` is
  `platform team`, a collective. None of the register's four `rota:` identifiers
  appears in it or in `docs/operations/alert-routing.md`, so the name above
  cannot be resolved to whoever is actually carrying the pager.
- **A local evaluator cannot announce its own disappearance.** The Hetzner
  Prometheus and Alertmanager remain co-located with the services, so their
  ordinary incident route ends with the host. The separately provisioned
  dead-man receiver is responsible for detecting that silence from outside the
  platform domain. Production deploy validates that the incident and dead-man
  endpoints are distinct external HTTPS receivers; a supervised missing-signal
  test and fresh receipt are still required before relying on it operationally.
- **Whoever declares must be reachable over something that does not run on the
  V1 host** — the host is the thing that is gone. See §2.
- **Record the detection instant.** `DETECTED_AT_EPOCH` is a real input to
  `scripts/isis/run_runpod_disaster_recovery_gameday.sh` and the field
  `DrillTiming.detectedAtMs` in `@oshun/workbench-kit/restore-consistency`
  exists for it. A recovery time measured from the moment somebody ran the
  restore command excludes detection and decision, which is most of a real
  recovery.

## 2. Communications

Nothing in the repository sends these; they are the operator's to send, and they
are listed with deadlines so that a missed one is a fact rather than an
impression.

| Audience  | Within | Content                                                                                    |
| --------- | ------ | ------------------------------------------------------------------------------------------ |
| Actors    | 15 min | that work is stopped, and that in-flight edits are not lost beyond the recovery point (§6) |
| Operators | 5 min  | that this runbook is running, and by whom                                                  |
| Providers | 60 min | only if a provider credential was on the lost host                                         |

**The channel must not be the estate, and the register now says so.**
`dep:notification-channel` is a target of `gd:regional-loss`: the sender is the
BFF (`apps/oshun/bff/src/auth/verification-email-sender.ts`), which is on the
lost host. Messages routed over it during a regional loss do not arrive, and
`runbookFaults` refuses a runbook that tries.

> **Gap, stated.** The estate has **no out-of-band channel implemented.** The
> three rows above travel over whatever the operator can reach from outside the
> box, and this repository does not provide it. That is the same class of gap as
> §8: a human control standing in for a mechanism.

## 3. Dependency assumptions

The recovery assumes exactly these, and nothing else:

- The restic repository is reachable and is **not** on the V1 host.
  `infra/hetzner/backup/offbox-backup.sh` refuses a local repository outright:
  `RESTIC_REPOSITORY must not be local to the V1 host`.
- Docker is available on the recovery host (`docker command is unavailable` is a
  hard failure in `box-loss-restore-drill.sh`).
- `find`, `jq`, `sha256sum` and `stat` are on `PATH` — the drill checks each.

It assumes nothing about the lost host. If a step needs something only the lost
host had, that step is wrong.

## 4. Recovery infrastructure

**A clean host with no running Oshun stack.** Not a dependency in the S10.5
register and never will be: it is provisioned for the recovery.

`box-loss-restore-drill.sh` enforces both halves of "clean":

```sh
docker ps --format '{{.Names}}' | grep -Eq '^oshun-(staging|production)(-|$)'
  # -> "an Oshun runtime is already running; this is not a clean-host drill"
find "${TARGET_DIR}" -mindepth 1 -maxdepth 1 -print -quit
  # -> "target directory is not empty"
```

`OSHUN_CLEAN_HOST_ID` is required and is recorded in the proof as
`targetHostId`, so the report says which host the estate came back on.

## 5. Secrets

Two, and both must be obtainable **without** the estate:

- `RESTIC_REPOSITORY` and the restic password (`RESTIC_PASSWORD` or
  `RESTIC_PASSWORD_FILE`), supplied in the recovery env file. The drill fails
  with `restic recovery password is absent` if neither is set.
- The recovery env file itself must deny group and other access; the drill
  refuses it otherwise
  (`recovery env permissions must deny group/other access`).

The drill's comment states the rule the rest of this runbook depends on: the
recovery env "is operator-controlled and **must not be fetched from the
repository being recovered**".

> **Closed on 2026-08-14 (S10.12.g, finding `B-2`).** This section used to read:
> "the restore proof asserts `keyRecoveryVerified: true` and there is no step in
> the script that exercises a key beyond the restic password itself." There is
> now: `infra/hetzner/backup/verify-restored-state.sh keys` checks the restored
> `stack.env` against **every `${VAR:?}` in `infra/hetzner/docker-compose.yml`**
> — the set the stack refuses to start without — and the drill writes
> `keyRecoveryVerified` as the value that check produced instead of as a
> literal. The required set is derived from the compose rather than listed, so a
> new required secret cannot be forgotten here. Retested both ways
> (`retest:key-recovery`): a complete environment is accepted, and one missing
> `MINIO_ROOT_PASSWORD` is refused by name.

## 6. Restore order

Run `infra/hetzner/backup/box-loss-restore-drill.sh`:

```sh
infra/hetzner/backup/box-loss-restore-drill.sh \
  production /path/to/recovery.env /path/to/empty-target /path/to/proof.json
```

It performs, in order: `restic check --read-data-subset` →
`restic restore latest --tag oshun-v1 --tag <stack>` → manifest schema check
(`oshun.v1-offbox-backup-manifest.v1`) → per-dump SHA-256 verification →
PostgreSQL into a scratch container → Redis → MinIO mirror → `stack.env`
presence.

**That order is safe only because nothing is serving traffic.** The drill runs
into an isolated docker network with no route in. When the estate is brought
back for actors, the order is the other way round for the stores that reference
each other:

| Bring back      | Before        | Because                                                                                                                                  |
| --------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| blob store      | primary store | artifact rows carry a blob key; a row exposed before its bytes is a dangling reference, and the bytes without a row are harmless garbage |
| primary store   | search index  | the index is projected from the rows and is rebuildable; the rows are not rebuildable from the index                                     |
| signing service | primary store | release rows carry the key version their signature was made under                                                                        |

This is the reverse of the CAPTURE order in
`@oshun/workbench-kit/restore-consistency`, and deliberately so: capture the
referring store first so inserts leave orphans, restore the referenced store
first so no row is exposed pointing at bytes that are not back yet.
`restoreOrderFaults` in `@oshun/workbench-kit/game-day` checks a proposed order
against the same reference graph.

**The full order, and it is longer than the three stores above.** A regional
loss takes ten registered dependencies, not four: `docker/docker-compose.yml`
declares the whole estate in one file with no placement constraint, so the
telemetry, the cache, the session authority and the notification sender go with
the data stores. Bring them back in this order:

1. `dep:identity` — nothing else is reachable by an actor until sessions are,
   and nothing in the list below is verifiable by anybody who cannot sign in.
2. `dep:telemetry-pipeline` — **before** the stores, not after. It is the
   instrument the rest of this restore is watched through, and a restore
   observed only by the operator's terminal is one whose failures are noticed in
   the order somebody happens to look.
3. `dep:notification-channel` — the sender, so §2's messages stop needing the
   out-of-band route the moment they can stop.
4. `dep:blob-store` — before the rows that reference it.
5. `dep:primary-store` — the rows.
6. `dep:read-cache` — after its source, and empty. A cache restored from a
   snapshot serves the estate's state as of the snapshot, which is the one
   failure in this list that looks like success.
7. `dep:primary-store-scan` — correct as soon as the rows are.
8. `dep:search-index` — start Qdrant empty and reproject personalization vectors
   from PostgreSQL; it is last because no Qdrant snapshot is authoritative.
9. `dep:job-queue`, then 10. `dep:job-worker-pool` — the pool rests on the
   queue, and a pool started first reports itself healthy while doing nothing.

`dep:signing-service` is not in the numbered list because a regional loss does
not take it: there is no signing service in the compose file, the signer is
in-process (see
[v1-dr-signing-service-outage.md](v1-dr-signing-service-outage.md)). If it is
ever restored alongside these, it goes **before** `dep:primary-store`, as the
third row of the table above says.

## 7. Validation

The drill's own floors, which are real checks and not assertions:

- every `*.dump` has a `.sha256` beside it and the bytes hash to it;
- `checksum_count > 0` — a snapshot containing no PostgreSQL dumps fails;
- `database_count == checksum_count` — every dump has metadata and every
  metadata has a dump;
- each restored database answers
  `SELECT count(*) >= 0 FROM pg_catalog.pg_class`;
- **Redis holds what the capture recorded.** The manifest carries
  `inventory.redisKeyCount` and `inventory.redisPersistentKeyCount`, taken from
  `INFO keyspace` at the instant of the forced `BGSAVE`, and
  `verify-restored-state.sh redis` refuses a restore that answers `PING` with an
  empty keyspace, one short of the persistent keys (keys with no TTL cannot
  expire, so a shortfall is loss), or one holding more than was captured (which
  would mean this is not the snapshot that was restored). A snapshot whose
  manifest predates those counts is refused rather than accepted on the `PING`.
- **MinIO holds what was mirrored**: `inventory.minioObjectCount` against the
  objects on the restored volume.
- `stack.env` carries every value the stack refuses to start without (§5).

> **Closed on 2026-08-14 (S10.12.g, finding `F-f2`).** This section used to
> read: "Redis, MinIO, the file state and the configuration are asserted `true`
> in the report from steps that an EMPTY store also passes." S10.12.f measured
> what that cost: `redis-server --appendonly yes` — how this estate runs it —
> creates a fresh AOF on start and never reads `dump.rdb`, so a host handed the
> checkpoint the backup forces and verifies comes up holding **0 of 25 entries**
> and answers `PONG`. The counts above are the repair, and `retest:F-f2` runs
> both arms: the pre-fix check accepts that empty restore, the repaired check
> refuses it by name, and the whole archived data directory is accepted at 25
> of 25.
>
> **Still open (`B-3a`):** the content-service and computer-use directories are
> checked for existence and nothing else. An empty one passes.

## 8. Fencing — ABSENT for a promoted primary

**Correction to an earlier version of this runbook, which said there is no
fencing anywhere in this estate.** That was a claim about a grep — for
`split-brain`, `fence`, `fencing` and `STONITH` across `docs/`, `infra/` and
`scripts/` — and the estate's one real fencing token is in none of those
directories and uses none of those words.

It is `live_media_pipeline_job_attempts.lease_token`, in
`libs/shared/live-media/src/sql-media-pipeline-job-store.ts`. Every heartbeat,
lease expiry and completion carries `AND lease_token = $4 AND status = 'leased'`
in its `UPDATE ... WHERE`, so a worker whose lease was reassigned updates **zero
rows** instead of overwriting the worker that replaced it. That is textbook
lease fencing, it works, and it is the pattern the rest of this section is
missing rather than a pattern the estate has never used.

**What is absent is a fence for a promoted primary**, which is what this
scenario needs: nothing rejects the writes of a host that comes back after its
replacement was promoted. A job attempt is protected from a stale worker; a
store is not protected from a stale host.

Consequence, stated so that it is a decision rather than an accident: if the
lost host returns to life while the recovery host is serving, there are two
primaries and nothing rejects the writes of the older one. Until a fencing token
covers the stores the way `lease_token` covers a job attempt, the recovery host
must not be brought up for actors while the original host might still be
reachable. That is a human control, and it is the weakest step in this runbook.

## 9. Failover — promoting the recovery host

Failover here is not a switch; it is the decision to let actors reach the
recovery host, and it is the step §8 makes dangerous.

1. Confirm the reconciler (§10) has run and returned no blockers. A host that is
   serving before reconciliation can repeat a completed provider effect.
2. Confirm the original host is unreachable **and** cannot come back on its own.
   With no fencing token, this is a human judgement and it is the one that
   causes split-brain when it is wrong.
3. Point DNS or the load balancer at `OSHUN_CLEAN_HOST_ID`. The recovery host is
   identified in the proof as `targetHostId`, so the record of where the estate
   went is the same artifact as the record that it came back.
4. Record the instant. Recovery time runs from the fault, not from this step.

`infra/hetzner/backup/box-loss-restore-drill.sh` deliberately does none of this:
it restores into an isolated docker network and refuses to run at all if an
Oshun runtime is present. Promotion is outside the drill and always has been.

## 10. Reconciliation — the restore is not finished when the restore finishes

The proof says so itself:

```json
{
  "status": "restore-pass-reconciliation-required",
  "safeToResumeAutonomy": false,
  "nextRequiredControl": "scripts/operations/v1-restore-reconcile.mjs"
}
```

Run `scripts/operations/v1-restore-reconcile.mjs` before resuming any autonomous
or effectful work. It compares restored provider effects, publications, queues,
invoices, artifacts, placements and approvals against what the outside world
observed, and returns blockers.

Before starting the HTTP-serving BFF, start PostgreSQL, Redis, and an empty
Qdrant, then run the bundled production reprojection entry point:

```bash
docker compose -p oshun-production \
  --env-file /opt/oshun/production/.env \
  -f /opt/oshun/bundle/docker-compose.yml \
  run --rm bff node dist/reproject-personalization-vector-index.js
```

The command creates or validates the collection and its `userId`,
`schemaVersion`, and `embeddingVersion` payload indexes, removes stale points,
projects every non-tombstoned PostgreSQL vector, and fails unless readback
count, payload version, and dimensions match. Starting BFF repeats the same
idempotent reconciliation before constructing HTTP. Production boot also
requires Redis, the deletion-attestation signer, and the independent remote
recovery-deletion journal; it replays every verified receipt before HTTP exists.
A missing dependency, failed reprojection, failed receipt import, or failed
replay keeps the BFF down. Do not route traffic around this admission boundary.

**Give it expected counts.** Every collection it is handed can be empty, and
empty reconciles clean — an empty restore produces the same verdict as a
complete one unless the expectation is supplied. Record what the estate held at
the recovery point, and compare.

**And the counts have to come from somewhere.** Nothing in this repository
writes that reconciler's input file;
`scripts/operations/v1-replay-inventory.mjs` reads the nine state classes out of
the database so there is an inventory to compare against, and refuses to report
a count for a store it could not read. The state-specific procedures for folding
each class back in — which work may be re-driven, which must never be, and how
the operator tells them apart — are in
[v1-reconciliation-and-replay.md](v1-reconciliation-and-replay.md). That
document is the rest of this step.

## 11. Failback

Returning to a rebuilt original host is the same procedure with the roles
swapped, plus one step this runbook is explicit about: **take a fresh backup of
the recovered estate before failing back.** The recovery host has been serving;
the snapshot it was built from has not been current since §6, and until a new
recovery point exists the estate's most recent backup predates the disaster.

## 12. What this runbook does not cover

- Partial restores (a single table, a single bucket prefix). See
  `database-backups.md`.
- A cluster that is still reachable — use the `pg_dump` path there; it is faster
  and does not need a clean host.
- Search and provenance bytes are not in the off-box backup set. The
  personalization Qdrant collection comes back only through the mandatory
  PostgreSQL reprojection above; other search recovery has its own runbook
  ([v1-dr-search-index-loss.md](v1-dr-search-index-loss.md)) and is likewise a
  reprojection, which is why derived indexes are not in the backup.
- **How any of this is rehearsed.** This runbook is what an operator does during
  an incident; [v1-game-day-plan.md](v1-game-day-plan.md) is how the estate is
  deliberately broken on a Tuesday so the steps here are tested before an
  incident tests them. It carries the fault controls, the reading that confirms
  each injection landed, the observers, the timed instants, the safety
  boundaries and the abort — and it records which five of the ten registered
  scenarios cannot be rehearsed on this estate at all, with the reason for each.
- **Whether the steps here actually work.**
  [v1-acceptance-matrix.md](v1-acceptance-matrix.md) is the result of running
  them: seven aspects — split-brain prevention, degraded operation, queued work,
  user communication, monitoring and alerts, the backup of the recovered
  primary, and the failback in §11 above — each answered by a step of one
  executed drill on a disposable estate. Three of the seven currently fail, and
  the matrix says which and by how much. Read it before trusting §8, §9 or §11.
- **What is being done about what those drills found.**
  [v1-follow-up-register.md](v1-follow-up-register.md) carries every finding the
  game-day programme measured — thirty of them — with a severity, an owner, a
  deadline and, for the ten that are closed, the run that closed them. Two of
  the closures repaired this runbook's own restore path: §5
  (`keyRecoveryVerified` was a literal) and §7 (a `PING` accepted an empty
  queue). The twenty still open are what an operator should expect to go wrong,
  ranked, with a date against each.

## 13. Machine-checkable statement

The block below is the input `runbookFaults` in `@oshun/workbench-kit/game-day`
grades, and `libs/oshun/workbench-kit/tools/runbook-scan.mjs` runs it against
the scenario register. It is here rather than in a test file so that the
statement the checker reads and the statement the operator reads are one object.

`promotesAStandby` is `true` and `fencing` is `null`, so the contract reports
`promotion-with-no-fencing` against this runbook. That refusal is **correct and
expected**: it is §8, stated in a form a machine can count.

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-regional-loss",
  "scenarioId": "gd:regional-loss",
  "version": "3",
  "detectionAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["out-of-band: a route that does not run on the V1 host"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["out-of-band: a route that does not run on the V1 host"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 300000
    },
    {
      "audience": "provider",
      "channelDependencyId": "out-of-band: a channel not hosted on the V1 box",
      "withinMs": 3600000
    }
  ],
  "assumes": [
    "the off-box restic repository, which offbox-backup.sh refuses to place on the V1 host",
    "docker on the recovery host",
    "find, jq, sha256sum and stat on PATH"
  ],
  "recoveryInfrastructureDependencyId": "a clean host with no running Oshun stack",
  "secrets": [
    {
      "id": "RESTIC_REPOSITORY and the restic password",
      "heldInDependencyId": "the operator-controlled recovery env file",
      "recoverableOutOfBand": true
    },
    {
      "id": "every other key on the lost host (keyRecoveryVerified is unproven, see section 5)",
      "heldInDependencyId": "the operator-controlled recovery env file",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": [
    "dep:identity",
    "dep:telemetry-pipeline",
    "dep:notification-channel",
    "dep:blob-store",
    "dep:primary-store",
    "dep:read-cache",
    "dep:primary-store-scan",
    "dep:search-index",
    "dep:job-queue",
    "dep:job-worker-pool"
  ],
  "fencing": null,
  "promotesAStandby": true,
  "validation": [
    "every *.dump has a .sha256 beside it and the bytes hash to it",
    "checksum_count > 0: a snapshot containing no PostgreSQL dumps fails",
    "database_count == checksum_count",
    "each restored database answers SELECT count(*) >= 0 FROM pg_catalog.pg_class",
    "the personalization Qdrant projection command reports matching authoritative, projected and verified counts",
    "BFF boot completes independent recovery-deletion receipt import and replay before HTTP exists",
    "v1-restore-reconcile.mjs returns no blockers against recorded expected counts"
  ],
  "failback": "the same procedure with the roles swapped, plus a fresh backup of the recovered estate taken before failing back: until a new recovery point exists the estate's most recent backup predates the disaster"
}
```