Status: encrypted off-box automation implemented; deployed backup, provider retention/immutability, independent key recovery, real box-loss drill, and measured production RPO/RTO remain unproved.
Implementation source commit: f6571b0089edb0fd374c4cf8a780ae9a7f1ea0dc.
Requirement disposition#
| OPS-002 requirement | Verdict | Evidence |
|---|---|---|
| Automate encrypted off-box backup for every required state class | Implemented | Canonical compose deploys a dedicated 21st GHCR image. The worker authenticates PostgreSQL dumps, forces Redis BGSAVE, exports MinIO through its API, and sends PostgreSQL, Redis, MinIO, content-service runs, computer-use snapshots, stack config/secrets, and the bundle to an encrypted restic repository. Backup credentials live in isolated backup.env, not application env_file. |
| Continuously prove checksums, retention, immutability, key recovery, and restore compatibility | Blocked on deployed evidence | The worker verifies dump SHA-256 and a configured remote data subset, emits checksummed receipts, and requires an immutability mode. Provider retention/object-lock state and independently recovered keys cannot be inferred from configuration; no authenticated deployed receipt exists. |
| Run a clean-host/box-loss restore drill | Blocked on a real snapshot and replacement host | The operator drill refuses an in-place/nonempty host, checks the encrypted repository, restores PostgreSQL/Redis/MinIO/file/config state into clean scratch services, measures RTO, and never starts autonomy. Its full control flow passes a hermetic Docker test, but no real production snapshot or replacement host was available. |
| Reconcile restored durable and external state before autonomy | Implemented control, unexecuted | The reconciliation gate covers publications, provider effects, queues, approvals, invoices, artifacts, placements, payload digests, external references, and idempotency keys. Restore proof explicitly sets safeToResumeAutonomy=false; a real inventory/result has not been run. |
| Prove completed mutating effects cannot repeat | Test-proved algorithm, production proof blocked | A restored-pending effect already completed at the provider is blocked for adoption, completed receipts must match remote payloads, unknown remote completions block, and pending work can resume only with its original key. No real post-restore provider reconciliation exists. |
| Record measured RPO/RTO and unresolved loss | Instrumented, production measurement blocked | Backup receipts record measured PostgreSQL age/RPO and elapsed time; restore proof records RTO. No deployed receipt/drill exists, so no production number is claimed. |
Backup boundary#
The worker is disabled by default. Enabling it requires all of:
- a non-local restic repository and named independent failure domain;
- restic repository encryption password or password file;
- restricted backend credentials held only in
/opt/oshun/<stack>/backup.env; - declared provider object lock, append-only rest server, or provider retention lock;
- explicit one-time repository initialization authority; and
- separate escrow of decryption and recovery credentials outside the V1 host.
Local paths, localhost rest servers, same-host failure-domain labels, implicit initialization, missing encryption keys, missing state roots, stale dumps, and checksum mismatch all fail before upload. Each successful run writes a checksummed receipt with snapshot ID, state/file/byte inventory, independent domain, immutability mode, retention target, measured source age, and duration. The receipt is observability evidence, not provider-side immutability proof.
Clean-host restore and autonomy fence#
backup/box-loss-restore-drill.sh requires a mode-600 recovery file and a
non-secret clean-host identity. It refuses to run beside an existing Oshun stack
or over a nonempty target. It then:
- verifies remote restic data with independently recovered credentials;
- restores the latest environment-tagged snapshot to an empty directory;
- authenticates every PostgreSQL dump and restores every database into a new pg16/pgvector container;
- boots the restored Redis state and imports the API-level MinIO export into a new object server;
- verifies content-service, computer-use, configuration, and secret state;
- records clean-host identity, source commit, snapshot, database count, and measured RTO; and
- returns
restore-pass-reconciliation-requiredwithsafeToResumeAutonomy=false.
The reconciliation engine returns pass only when every restore proof is true,
every restored/external collection agrees, approvals remain current, completed
effect receipts and payloads agree, no effect key is duplicated, and no remote
completion lacks a durable receipt. A pending local effect that is already
complete remotely is never retryable; it must first be adopted into durable
state. The result retains the explicit rule to keep OSHUN_AUTONOMY_AUTOCYCLE=0
until reconciliation passes.
Verification#
pnpm run verify:operations:v1-disaster-recovery
node:test: 12/12 passed
node --test scripts/operations/v1-topology-record.test.mjs
node:test: 8/8 passed
docker build -f deploy/v1/hetzner/backup/Dockerfile.offbox ...
passed
restic 0.18.1
mc RELEASE.2025-08-13T08-35-41Z
redis-cli 7.4.9
jq 1.7.1
docker compose ... config --quiet
passed with a synthetic non-secret stack env
ESLint, Prettier, sh -n, bash -n, and git diff --check
passed
Adversarial coverage includes same-host repositories, absent encryption keys, disabled operation, corrupt/stale dumps, unauthorized repository creation, in-place restore, weak recovery-file permissions, nonempty targets, incomplete restore proof, duplicate effect keys, unknown remote completions, provider/local payload conflict, expired approvals, publication drift, and the critical pending-locally/completed-remotely replay case.
Remaining external evidence#
The unchecked ledger items require actual operator authority and infrastructure: a configured remote repository, provider retention/object-lock evidence, independent recovery credentials, an authenticated backup receipt, a truly clean replacement host, live external inventories, and a completed restore/reconcile record. Until those exist, there is no claim of achieved retention, immutability, key recovery, restore compatibility, RPO, RTO, or replay-free production recovery.