Status: repository topology, failure domains, and recovery objectives verified; live-production topology and second-domain failover blocked on operator evidence.
The machine-verifiable record is
OPS-001/repository-topology.json. It was
generated from source commit f6571b0089edb0fd374c4cf8a780ae9a7f1ea0dc and has
payload SHA-256
9bbc8e1cc87f75321c27fe3dc62cb46d8f630898988cf5c6feb43bf24a8d49a1.
Requirement disposition#
| OPS-001 requirement | Verdict | Evidence |
|---|---|---|
| Generate a canonical repository topology record | Pass | The record hashes 25 canonical compose, Caddy, workflow, deployment, backup/restore, cycle-driver, environment-surface, README, generator, and policy sources. CI and the local verifier reject source, inventory, route, volume, workflow, claim, or checksum drift. |
| Probe authorized live production | Blocked | No authorized target identity, production hostname, SSH/provider credential, or monitoring endpoint is available in the repository or execution environment. The checked-in production Caddy site is named production.caddy.disabled, so the active *.caddy import does not load it. No live claim is made. |
| Enumerate shared failure domains | Pass | Seven explicit domains cover the host, rootful Docker daemon, kernel/disk/IP, single data instances, single edge, process-local cycle driver, and same-disk backup placement. |
| Define state-class objectives | Pass | Nine state classes define RPO, RTO, availability, capacity, maintenance, recovery procedure, and owner. They are objectives, not claims that production currently meets them. |
| Implement and prove a second failure domain | Blocked | Canonical sources declare one host and no replica/quorum/traffic-failover path. No authenticated failover record exists. Whole-box backup suggestions are not treated as resilience. |
Future evidence is intentionally separated from repository inference. An
authorized probe must be recorded at
evidence/v1-v9/OPS-001/live-production-probe.json; authenticated cross-domain
failover evidence must be recorded at
evidence/v1-v9/OPS-001/second-failure-domain-proof.json. The generator
validates their schemas and refuses to promote either requirement without
complete proof.
Declared topology#
- One Hetzner CCX23 is the shared host for the declared staging stack and the optional production stack. Both share one kernel, NVMe disk, public IP, rootful Docker daemon, and Caddy edge.
- Compose declares 26 services: 20 application images, a GHCR-built off-box backup worker, PostgreSQL, Redis, MinIO, the one-shot MinIO initializer, and local PostgreSQL backup. Steady state has 6,400 MiB of limits; the initializer adds a 64 MiB startup ceiling, making the literal all-service total 6,464 MiB.
- Six local named volumes hold PostgreSQL, Redis, MinIO, content-service run state, Psyche computer-use snapshots, and backup-worker scratch state. Run/review/publication snapshots used by the BFF are placed in the single PostgreSQL instance.
- One Caddy container exposes ten host routes on 80/443. PostgreSQL alone has a host-published application port and it is bound to loopback.
- The deployment workflow builds immutable images in GHCR, rsyncs the bundle, and invokes the environment-locked SSH deployment path. This is a deployment mechanism, not a redundant runtime control plane.
- The BFF process starts its autonomy cycle driver by default when a director is
bound. No distributed lease is declared; a future secondary must keep
OSHUN_AUTONOMY_AUTOCYCLE=0until promotion or implement leader election, or multiple BFFs can schedule duplicate cycles. - PostgreSQL dumps cover every non-template database, carry SHA-256 and metadata, retain 14 days by default, and run at startup plus cron. A disabled-by-default restic worker now declares encrypted independent-domain coverage for those dumps, Redis, MinIO API exports, file run/snapshot state, and configuration. This is implementation evidence only: no deployed receipt, remote retention proof, or clean-host drill is claimed.
Declared external dependencies are Hetzner host/storage/network, DNS and ACME, GitHub Actions/GHCR, an operator-supplied off-box backup repository, OpenRouter, optional RunPod GPU inference, the currently external/missing Metis Python backend, and an undeclared monitoring plane. The record keeps these repository observations distinct from any future live probe.
Failure-domain consequences#
| Shared domain | Consequence |
|---|---|
| CCX23 host | One host loss stops BFF/director, all other runtime, every data instance, backups, and edge together. |
| Docker daemon/kernel/disk/IP | A daemon, kernel, disk, or network failure crosses both environment projects and all state classes. |
| PostgreSQL/Redis/MinIO singletons | There is no declared replica, quorum, promotion, or fencing mechanism for authoritative, operational, or object state. |
| Caddy edge | One container/address/certificate-state path serves every public route. |
| BFF cycle driver | BFF loss stops scheduling; naive active-active BFF scale-out risks multiple schedulers. |
| Backup colocation | Whole-box loss removes local dumps/receipts; the remote snapshot survives only after enablement and a passing receipt. |
The full recovery-objective matrix is canonical in the JSON policy and record. The most stringent targets are Redis operational state at 1-minute RPO/15-minute RTO, PostgreSQL authoritative state at 5-minute RPO/30-minute RTO, and edge routing at 15-minute RTO/99.95% monthly availability. Each target also states capacity headroom, maintenance constraints, a recovery procedure, and ownership.
Verification#
The focused gate runs eight adversarial tests and a source-sensitive record check:
pnpm run verify:operations:v1-topology-record
node:test: 8/8 passed
record: 26 services, 10 routes
live topology: blocked
second failure domain: blocked
pnpm exec eslint scripts/operations/v1-topology-record.mjs \
scripts/operations/v1-topology-record.test.mjs --no-ignore --max-warnings=0
passed
pnpm exec prettier --check <OPS-001 changed files>
passed
git diff --check
passed
The tests remove/rename a compose service, move a durable-volume consumer, change the GHCR workflow inventory, mutate a Caddy upstream, tamper with the record, and change one byte of canonical source. They also assert that same-host backup and repository-only evidence remain blocked rather than becoming false live/resilience claims.