# Runbook: V1 Relational Corruption (`gd:relational-corruption`)

Owner: workbench-platform on-call (`rota:workbench-platform`). Last reviewed:
2026-08-14. Version: 1.

> The relational store answers, and the rows are wrong. Corruption rather than
> absence: every liveness check passes, the recovery is a restore to a recovery
> point rather than a restart, and putting the dependency back does not undo it.

**This is the scenario the estate is least equipped to notice.** A store that is
down announces itself. A store that returns wrong rows does not, and there is no
scheduled integrity check on this estate: a search across `scripts/`, `infra/`
and `.github/` for `pg_amcheck`, a row-checksum sweep or an invariant scan
returns nothing. Detection here is a report from an actor or a reconciliation
that refuses, and both of those come after the damage has been read.

Related: [database-backups.md](database-backups.md) documents the `pg_dump` path
this recovery uses; it does not mention that a V1 restore is unfinished until
the reconciler runs (§10).

## 1. Detection and declaration

**Detection signals that exist.**

- `scripts/operations/v1-restore-reconcile.mjs` compares restored provider
  effects, publications, queues, invoices, artifacts, placements and approvals
  against what the outside world observed, and returns blockers. It is written
  for the post-restore path but it is the only thing on this estate that
  compares stored state to an independent observation.
- `scripts/db-backup.sh` writes a `manifest.txt` of SHA-256 digests and
  `scripts/db-restore.sh` aborts with exit code 2 when a digest disagrees. That
  detects corruption **of the backup**, not of the live store.
- **PostgreSQL itself refuses a GARBLED page and cannot see a ZEROED one, and
  the difference decides what detection is possible here.** Measured on
  2026-08-14 over four arms (`retest:D-F4` in
  `scripts/operations/v1-follow-up-retest.mjs`), same table, same 2000 rows, one
  page overwritten:

  | page data | `data_checksums` | what a `count(*)` returned        |
  | --------- | ---------------- | --------------------------------- |
  | random    | off              | `ERROR: invalid page in block 1`  |
  | random    | on               | the same ERROR, plus the checksum |
  | zeros     | off              | **1815 of 2000 rows, no error**   |
  | zeros     | on               | **1815 of 2000 rows, no error**   |

  An all-zero page is a valid unused page to PostgreSQL, so a store that lost
  one answers cheerfully with fewer rows.
  `POSTGRES_INITDB_ARGS: --data-checksums` is now set in
  `infra/hetzner/docker-compose.yml` and it buys the diagnosis on the first two
  rows, not detection on the last two — and only for clusters initialised after
  that change, which means every clean-host restore and not the running
  production volume. **Detection of a zeroed page has to come from a row count
  compared against something outside the store**, which is what the reconciler
  in §10 is for.

- Detection authority: **workbench-platform on-call**, contact
  `rota:workbench-platform`.
- Declaration authority: **production operations duty officer**, contact
  `rota:prod-ops`.
- **Gap, stated.** `docs/operations/on-call.md` describes a weekly
  primary/secondary rotation and names no person and no rotation config; none of
  the register's `rota:` identifiers appears in it or in
  `docs/operations/alert-routing.md`. The name above is registered; the route
  from it to whoever is carrying the pager is not.
- **Declare early and read-only first.** Because the store answers, every minute
  between onset and declaration is a minute of reads returning wrong answers and
  writes built on them. The first action is to stop writes, not to diagnose.

## 2. Communications

| Audience  | Within | Content                                                                                          |
| --------- | ------ | ------------------------------------------------------------------------------------------------ |
| Actors    | 15 min | that work between the recovery point and now may be rolled back, and which surfaces are affected |
| Operators | 5 min  | that this runbook is running, and by whom                                                        |

Both travel over `dep:notification-channel`, which this scenario leaves
standing.

## 3. Dependency assumptions

- `dep:blob-store` is healthy and is **not** rolled back with the relational
  store. Restoring rows to an earlier point while the bytes stay current leaves
  blobs with no row pointing at them — harmless garbage in the direction the
  reference graph runs (`dep:primary-store` → `dep:blob-store`).
- A backup set exists whose manifest verifies. `db-restore.sh` refuses to run
  otherwise, which is the correct behaviour and also means an unverifiable
  backup turns this incident into a data-loss incident.

## 4. Recovery infrastructure

**In place, and that is a choice with a cost.** The restore runs on the same
host. Restoring over the corrupt database destroys the evidence of what went
wrong, so restore into a scratch database first
(`scripts/db-restore.sh BACKUP_DIR --databases <name>` against a renamed target)
when the cause is unknown and the corruption is not spreading.

## 5. Secrets

- PostgreSQL credentials (`PG_HOST`, `PG_PORT`, `PG_USER`, `PGPASSWORD` — the
  same variables `db-backup.sh` uses). Held in the operator's environment, not
  fetched from the estate being repaired.

Both are recoverable out of band: the `configuration-and-secrets` state class is
in the off-box backup (`infra/hetzner/backup/offbox-backup.sh`).

## 6. Restore order

1. `dep:primary-store` — restore the rows. `scripts/db-restore.sh` verifies
   every SHA-256 in `manifest.txt` before invoking `pg_restore` and aborts on
   any mismatch.
2. `dep:primary-store-scan` — the scan standby reads the same rows and is
   correct as soon as they are.
3. `dep:search-index` — reproject last. The index is built from the rows
   (`dep:search-index` → `dep:primary-store` in the reference graph), so an
   index rebuilt from corrupt rows is corrupt, and one rebuilt before the
   restore is stale in a way that looks healthy.

`dep:blob-store` is not in this order because it is not taken away; if it were,
it would come **first**, because a row exposed before its bytes is a dangling
reference.

> **There is no point-in-time recovery on this estate.** `archive_mode = on`
> appears in `infra/yemaya/postgres-ha/primary.conf`, and nothing outside
> `infra/yemaya/postgres-ha/` references that directory — the V1 compose does
> not use it. The recovery point is therefore the last `pg_dump`, and the
> interval between dumps is the exposure. You cannot restore to the instant
> before the corruption; you restore to the last backup that verifies.

## 7. Validation

- `db-restore.sh` manifest verification (exit code 2 on mismatch) — real, and it
  runs before any data is written.
- `SELECT count(*) >= 0 FROM pg_catalog.pg_class` per restored database, the
  floor `box-loss-restore-drill.sh` applies.
- **A named count.** Record what the estate held at the recovery point and
  compare. Every collection the reconciler is handed can be empty, and empty
  reconciles clean — an empty restore produces the same verdict as a complete
  one unless the expectation is supplied.
- The invariant that failed. Corruption is specific: whatever wrong answer
  raised this incident is the query to re-run.

## 8. Fencing — NOT APPLICABLE

Nothing is promoted. There is one primary before and after, so there is no
window in which two of them are writing. The dangerous window here is a
different one: between declaration and the stopping of writes, the application
is writing rows derived from wrong reads. Stopping writes is §1's first action
for that reason.

## 9. Failover

There is nothing to fail over to. `infra/yemaya/postgres-ha/failover.sh` exists
and promotes a standby; the V1 compose has one `postgres` service and no
replica, so the script's own `check_standby` refuses with "This server is not in
recovery mode (not a standby)". See
[v1-dr-database-loss.md](v1-dr-database-loss.md) §9 for the detail — it is the
same absent standby.

## 10. Reconciliation — the restore is not finished when the restore finishes

Run `scripts/operations/v1-restore-reconcile.mjs` before resuming any autonomous
or effectful work. A rollback to a recovery point un-does rows the outside world
already saw: a publication that went out, a provider call that was charged, an
invoice that was issued. The reconciler compares those against what was observed
and returns blockers.

This matters more here than after a box loss. A box loss stops the estate before
it can act on wrong data; corruption lets it keep acting.

## 11. Failback

There is no alternate topology to return from. Failback is the resumption of
writes, and it has one precondition: the reconciler returns no blockers. Take a
fresh backup immediately afterwards — the restored estate's most recent backup
is, by construction, the one it was restored from, and it predates the incident.

## 12. What this runbook does not cover

- A store that is simply gone. That is
  [v1-dr-database-loss.md](v1-dr-database-loss.md); the recovery point there is
  untouched.
- Logical corruption confined to one table. `database-backups.md` covers partial
  restores and is faster.
- Detecting corruption. Nothing on this estate does, and this runbook will not
  pretend a detector exists.

## 13. Machine-checkable statement

<!-- oshun:runbook -->

```json
{
  "id": "runbook:v1-relational-corruption",
  "scenarioId": "gd:relational-corruption",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:blob-store"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "PGPASSWORD",
      "heldInDependencyId": "dep:primary-store",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": [
    "dep:primary-store",
    "dep:primary-store-scan",
    "dep:search-index"
  ],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "db-restore.sh verifies every SHA-256 in manifest.txt and aborts with exit code 2 on mismatch",
    "each restored database answers SELECT count(*) >= 0 FROM pg_catalog.pg_class",
    "recorded counts at the recovery point are compared against the restored counts",
    "the query whose wrong answer raised the incident returns the right one"
  ],
  "failback": "resume writes once v1-restore-reconcile.mjs returns no blockers, then take a fresh backup: the restored estate's newest backup is the one it was restored from"
}
```