Runbook: V1 Blob Store Loss (gd:blob-store-loss)

Owner: workbench-platform on-call (rota:workbench-platform). Last reviewed: 2026-08-14. Version: 1.

Rows survive and the bytes they point at do not. The reference direction decides the damage: the relational store holds keys into the blob store, so this is the outage that produces dangling references rather than orphans.

Related: object-storage-backups.md covers S3 versioning and cross-region replication for the multi-domain estate's oshun-prod-uploads buckets. It does not cover this. The V1 blob store is MinIO on the V1 host, and its only capture is the mc mirror inside infra/hetzner/backup/offbox-backup.sh.

1. Detection and declaration#

Detection signal. Eight capabilities stop and they stop in a distinctive pattern: reads of items with media fail while reads of the same items without media succeed. slo:item-read-availability, slo:render-success-rate, slo:export-success-rate and slo:save-durability all rest on it.

The clearest single observable is that a row resolves and its blob key 404s — which is the dangling reference the reference graph predicts and the reason this scenario is not symmetric with a relational loss.

• Detection authority: workbench-platform on-call, contact rota:workbench-platform.
• Declaration authority: production operations duty officer, contact rota:prod-ops.
• Gap, stated. docs/operations/on-call.md describes a weekly primary/secondary rotation and names no person and no rotation config; none of the register's rota: identifiers appears in it or in docs/operations/alert-routing.md. The name above is registered; the route from it to whoever is carrying the pager is not.

2. Communications#

Both travel over dep:notification-channel, which this scenario leaves standing.

3. Dependency assumptions#

• dep:primary-store is healthy and must not be rolled back. The rows hold the blob keys and the recorded digests, and they are the only index of what the blob store should contain. Restoring bytes without them is a bucket of unaddressable objects.
• dep:job-queue is healthy: renders and exports that failed during the outage are re-run through it.

4. Recovery infrastructure#

In place. MinIO is a service in docker/docker-compose.yml. The restore is a mirror back into it.

5. Secrets#

• MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, the credentials offbox-backup.sh uses for mc alias set v1-source.
• The restic repository and its password, if the bytes come from the off-box snapshot rather than from a surviving local copy.

Both recoverable out of band: the restic repository is required to be off the V1 host — offbox-backup.sh refuses a local one outright with RESTIC_REPOSITORY must not be local to the V1 host.

6. Restore order#

dep:blob-store comes back before anything that references it. The reference graph runs dep:primary-store → dep:blob-store (artifact rows carry a blob key and a recorded digest), so:

This is the reverse of the capture order, and deliberately so: capture the referring store first so inserts in the skew window leave orphans, restore the referenced store first so no row is exposed pointing at bytes that are not back. restoreOrderFaults in @oshun/workbench-kit/game-day checks a proposed order against the same graph.

In practice the relational store is not down in this scenario, which means the rows are already exposed and pointing at bytes that are not back. That is the dangling-reference window, it opens at the fault and not at the restore, and nothing closes it except finishing §7.

7. Validation#

The rows carry a recorded digest, and that is what makes this store verifiable where Redis and the file state are not:

• read back three known objects by digest and compare against the digest in the artifact row — not merely that the key resolves;
• the object count is compared against the count of artifact rows with a blob key. Record it before the restore or take it from the relational store, which survived.

Gap, stated. box-loss-restore-drill.sh asserts its MinIO step from an mc mirror that an empty store also passes. Only PostgreSQL has a count floor in that drill. Do not read a successful mirror as evidence that the contents came back.

8. Fencing — NOT APPLICABLE#

Nothing is promoted; there is one blob store before and after. The adjacent risk is not split-brain but write loss during the window: uploads attempted while the store is down must be refused rather than acknowledged, or the rows will record artifacts whose bytes were never written — a dangling reference the restore cannot fix, because the bytes never existed.

9. Failover#

There is no second blob store to fail over to. The V1 estate has one MinIO instance, and the cross-region replication described in object-storage-backups.md belongs to the S3 deployment, not to this one.

10. Reconciliation#

Run scripts/operations/v1-restore-reconcile.mjs. Artifacts are among the collections it compares against what the outside world observed, and this scenario is the one where its artifact reconciliation earns its place: a published artifact whose bytes are gone was seen by somebody outside the estate.

Give it expected counts. An empty collection reconciles clean, and after a blob loss "no artifacts" is exactly the wrong answer to accept quietly.

11. Failback#

The estate never left its normal topology. Failback is the confirmation that the dangling-reference window is closed: every artifact row with a blob key resolves to bytes whose digest matches. Until that holds, the estate is serving rows that point at nothing.

Take a fresh backup afterwards — the mirror the restore came from is now the estate's most recent capture and it predates the incident.

12. What this runbook does not cover#

• The multi-domain S3 estate. See object-storage-backups.md; versioning and cross-region replication are real there and absent here.
• Per-object recovery of a single deleted key, which that runbook also covers and which does not need this procedure.
• A total host loss, which takes MinIO and the rows together — see v1-disaster-recovery.md.

13. Machine-checkable statement#

{
  "id": "runbook:v1-blob-store-loss",
  "scenarioId": "gd:blob-store-loss",
  "version": "1",
  "detectionAuthority": {
    "role": "service on-call",
    "name": "workbench-platform on-call",
    "reachableOver": ["dep:notification-channel"]
  },
  "declarationAuthority": {
    "role": "operations duty officer",
    "name": "production operations duty officer",
    "reachableOver": ["dep:notification-channel"]
  },
  "communications": [
    {
      "audience": "actors",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 900000
    },
    {
      "audience": "operators",
      "channelDependencyId": "dep:notification-channel",
      "withinMs": 300000
    }
  ],
  "assumes": ["dep:primary-store", "dep:job-queue"],
  "recoveryInfrastructureDependencyId": null,
  "secrets": [
    {
      "id": "MINIO_ROOT_PASSWORD",
      "heldInDependencyId": "dep:blob-store",
      "recoverableOutOfBand": true
    },
    {
      "id": "restic repository password",
      "heldInDependencyId": "dep:blob-store",
      "recoverableOutOfBand": true
    }
  ],
  "restoreOrder": ["dep:blob-store"],
  "fencing": null,
  "promotesAStandby": false,
  "validation": [
    "three known objects read back by digest and compared against the digest recorded in the artifact row",
    "the object count matches the count of artifact rows carrying a blob key"
  ],
  "failback": "confirm every artifact row with a blob key resolves to bytes whose digest matches, then take a fresh backup"
}