# OPS-002 — V1 backup, box-loss restore, and reconciliation

Status: **encrypted off-box automation implemented; deployed backup, provider
retention/immutability, independent key recovery, real box-loss drill, and
measured production RPO/RTO remain unproved**.

Implementation source commit: `f6571b0089edb0fd374c4cf8a780ae9a7f1ea0dc`.

## Requirement disposition

| OPS-002 requirement                                                                            | Verdict                                         | Evidence                                                                                                                                                                                                                                                                                                                                                                                      |
| ---------------------------------------------------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Automate encrypted off-box backup for every required state class                               | Implemented                                     | Canonical compose deploys a dedicated 21st GHCR image. The worker authenticates PostgreSQL dumps, forces Redis BGSAVE, exports MinIO through its API, and sends PostgreSQL, Redis, MinIO, content-service runs, computer-use snapshots, stack config/secrets, and the bundle to an encrypted restic repository. Backup credentials live in isolated `backup.env`, not application `env_file`. |
| Continuously prove checksums, retention, immutability, key recovery, and restore compatibility | Blocked on deployed evidence                    | The worker verifies dump SHA-256 and a configured remote data subset, emits checksummed receipts, and requires an immutability mode. Provider retention/object-lock state and independently recovered keys cannot be inferred from configuration; no authenticated deployed receipt exists.                                                                                                   |
| Run a clean-host/box-loss restore drill                                                        | Blocked on a real snapshot and replacement host | The operator drill refuses an in-place/nonempty host, checks the encrypted repository, restores PostgreSQL/Redis/MinIO/file/config state into clean scratch services, measures RTO, and never starts autonomy. Its full control flow passes a hermetic Docker test, but no real production snapshot or replacement host was available.                                                        |
| Reconcile restored durable and external state before autonomy                                  | Implemented control, unexecuted                 | The reconciliation gate covers publications, provider effects, queues, approvals, invoices, artifacts, placements, payload digests, external references, and idempotency keys. Restore proof explicitly sets `safeToResumeAutonomy=false`; a real inventory/result has not been run.                                                                                                          |
| Prove completed mutating effects cannot repeat                                                 | Test-proved algorithm, production proof blocked | A restored-pending effect already completed at the provider is blocked for adoption, completed receipts must match remote payloads, unknown remote completions block, and pending work can resume only with its original key. No real post-restore provider reconciliation exists.                                                                                                            |
| Record measured RPO/RTO and unresolved loss                                                    | Instrumented, production measurement blocked    | Backup receipts record measured PostgreSQL age/RPO and elapsed time; restore proof records RTO. No deployed receipt/drill exists, so no production number is claimed.                                                                                                                                                                                                                         |

## Backup boundary

The worker is disabled by default. Enabling it requires all of:

- a non-local restic repository and named independent failure domain;
- restic repository encryption password or password file;
- restricted backend credentials held only in `/opt/oshun/<stack>/backup.env`;
- declared provider object lock, append-only rest server, or provider retention
  lock;
- explicit one-time repository initialization authority; and
- separate escrow of decryption and recovery credentials outside the V1 host.

Local paths, localhost rest servers, same-host failure-domain labels, implicit
initialization, missing encryption keys, missing state roots, stale dumps, and
checksum mismatch all fail before upload. Each successful run writes a
checksummed receipt with snapshot ID, state/file/byte inventory, independent
domain, immutability mode, retention target, measured source age, and duration.
The receipt is observability evidence, not provider-side immutability proof.

## Clean-host restore and autonomy fence

`backup/box-loss-restore-drill.sh` requires a mode-600 recovery file and a
non-secret clean-host identity. It refuses to run beside an existing Oshun stack
or over a nonempty target. It then:

1. verifies remote restic data with independently recovered credentials;
2. restores the latest environment-tagged snapshot to an empty directory;
3. authenticates every PostgreSQL dump and restores every database into a new
   pg16/pgvector container;
4. boots the restored Redis state and imports the API-level MinIO export into a
   new object server;
5. verifies content-service, computer-use, configuration, and secret state;
6. records clean-host identity, source commit, snapshot, database count, and
   measured RTO; and
7. returns `restore-pass-reconciliation-required` with
   `safeToResumeAutonomy=false`.

The reconciliation engine returns pass only when every restore proof is true,
every restored/external collection agrees, approvals remain current, completed
effect receipts and payloads agree, no effect key is duplicated, and no remote
completion lacks a durable receipt. A pending local effect that is already
complete remotely is never retryable; it must first be adopted into durable
state. The result retains the explicit rule to keep `OSHUN_AUTONOMY_AUTOCYCLE=0`
until reconciliation passes.

## Verification

```text
pnpm run verify:operations:v1-disaster-recovery
  node:test: 12/12 passed

node --test scripts/operations/v1-topology-record.test.mjs
  node:test: 8/8 passed

docker build -f deploy/v1/hetzner/backup/Dockerfile.offbox ...
  passed
  restic 0.18.1
  mc RELEASE.2025-08-13T08-35-41Z
  redis-cli 7.4.9
  jq 1.7.1

docker compose ... config --quiet
  passed with a synthetic non-secret stack env

ESLint, Prettier, sh -n, bash -n, and git diff --check
  passed
```

Adversarial coverage includes same-host repositories, absent encryption keys,
disabled operation, corrupt/stale dumps, unauthorized repository creation,
in-place restore, weak recovery-file permissions, nonempty targets, incomplete
restore proof, duplicate effect keys, unknown remote completions, provider/local
payload conflict, expired approvals, publication drift, and the critical
pending-locally/completed-remotely replay case.

## Remaining external evidence

The unchecked ledger items require actual operator authority and infrastructure:
a configured remote repository, provider retention/object-lock evidence,
independent recovery credentials, an authenticated backup receipt, a truly clean
replacement host, live external inventories, and a completed restore/reconcile
record. Until those exist, there is no claim of achieved retention,
immutability, key recovery, restore compatibility, RPO, RTO, or replay-free
production recovery.
