Eve library

OPS-001 — V1 production-topology verification

Evidence

Revision 1

On this page

Status: repository topology, failure domains, and recovery objectives verified; live-production topology and second-domain failover blocked on operator evidence.

The machine-verifiable record is OPS-001/repository-topology.json. It was generated from source commit f6571b0089edb0fd374c4cf8a780ae9a7f1ea0dc and has payload SHA-256 9bbc8e1cc87f75321c27fe3dc62cb46d8f630898988cf5c6feb43bf24a8d49a1.

Requirement disposition#

OPS-001 requirement Verdict Evidence
Generate a canonical repository topology record Pass The record hashes 25 canonical compose, Caddy, workflow, deployment, backup/restore, cycle-driver, environment-surface, README, generator, and policy sources. CI and the local verifier reject source, inventory, route, volume, workflow, claim, or checksum drift.
Probe authorized live production Blocked No authorized target identity, production hostname, SSH/provider credential, or monitoring endpoint is available in the repository or execution environment. The checked-in production Caddy site is named production.caddy.disabled, so the active *.caddy import does not load it. No live claim is made.
Enumerate shared failure domains Pass Seven explicit domains cover the host, rootful Docker daemon, kernel/disk/IP, single data instances, single edge, process-local cycle driver, and same-disk backup placement.
Define state-class objectives Pass Nine state classes define RPO, RTO, availability, capacity, maintenance, recovery procedure, and owner. They are objectives, not claims that production currently meets them.
Implement and prove a second failure domain Blocked Canonical sources declare one host and no replica/quorum/traffic-failover path. No authenticated failover record exists. Whole-box backup suggestions are not treated as resilience.

Future evidence is intentionally separated from repository inference. An authorized probe must be recorded at evidence/v1-v9/OPS-001/live-production-probe.json; authenticated cross-domain failover evidence must be recorded at evidence/v1-v9/OPS-001/second-failure-domain-proof.json. The generator validates their schemas and refuses to promote either requirement without complete proof.

Declared topology#

  • One Hetzner CCX23 is the shared host for the declared staging stack and the optional production stack. Both share one kernel, NVMe disk, public IP, rootful Docker daemon, and Caddy edge.
  • Compose declares 26 services: 20 application images, a GHCR-built off-box backup worker, PostgreSQL, Redis, MinIO, the one-shot MinIO initializer, and local PostgreSQL backup. Steady state has 6,400 MiB of limits; the initializer adds a 64 MiB startup ceiling, making the literal all-service total 6,464 MiB.
  • Six local named volumes hold PostgreSQL, Redis, MinIO, content-service run state, Psyche computer-use snapshots, and backup-worker scratch state. Run/review/publication snapshots used by the BFF are placed in the single PostgreSQL instance.
  • One Caddy container exposes ten host routes on 80/443. PostgreSQL alone has a host-published application port and it is bound to loopback.
  • The deployment workflow builds immutable images in GHCR, rsyncs the bundle, and invokes the environment-locked SSH deployment path. This is a deployment mechanism, not a redundant runtime control plane.
  • The BFF process starts its autonomy cycle driver by default when a director is bound. No distributed lease is declared; a future secondary must keep OSHUN_AUTONOMY_AUTOCYCLE=0 until promotion or implement leader election, or multiple BFFs can schedule duplicate cycles.
  • PostgreSQL dumps cover every non-template database, carry SHA-256 and metadata, retain 14 days by default, and run at startup plus cron. A disabled-by-default restic worker now declares encrypted independent-domain coverage for those dumps, Redis, MinIO API exports, file run/snapshot state, and configuration. This is implementation evidence only: no deployed receipt, remote retention proof, or clean-host drill is claimed.

Declared external dependencies are Hetzner host/storage/network, DNS and ACME, GitHub Actions/GHCR, an operator-supplied off-box backup repository, OpenRouter, optional RunPod GPU inference, the currently external/missing Metis Python backend, and an undeclared monitoring plane. The record keeps these repository observations distinct from any future live probe.

Failure-domain consequences#

Shared domain Consequence
CCX23 host One host loss stops BFF/director, all other runtime, every data instance, backups, and edge together.
Docker daemon/kernel/disk/IP A daemon, kernel, disk, or network failure crosses both environment projects and all state classes.
PostgreSQL/Redis/MinIO singletons There is no declared replica, quorum, promotion, or fencing mechanism for authoritative, operational, or object state.
Caddy edge One container/address/certificate-state path serves every public route.
BFF cycle driver BFF loss stops scheduling; naive active-active BFF scale-out risks multiple schedulers.
Backup colocation Whole-box loss removes local dumps/receipts; the remote snapshot survives only after enablement and a passing receipt.

The full recovery-objective matrix is canonical in the JSON policy and record. The most stringent targets are Redis operational state at 1-minute RPO/15-minute RTO, PostgreSQL authoritative state at 5-minute RPO/30-minute RTO, and edge routing at 15-minute RTO/99.95% monthly availability. Each target also states capacity headroom, maintenance constraints, a recovery procedure, and ownership.

Verification#

The focused gate runs eight adversarial tests and a source-sensitive record check:

text
pnpm run verify:operations:v1-topology-record
  node:test: 8/8 passed
  record: 26 services, 10 routes
  live topology: blocked
  second failure domain: blocked

pnpm exec eslint scripts/operations/v1-topology-record.mjs \
  scripts/operations/v1-topology-record.test.mjs --no-ignore --max-warnings=0
  passed

pnpm exec prettier --check <OPS-001 changed files>
  passed

git diff --check
  passed

The tests remove/rename a compose service, move a durable-volume consumer, change the GHCR workflow inventory, mutate a Caddy upstream, tamper with the record, and change one byte of canonical source. They also assert that same-host backup and repository-only evidence remain blocked rather than becoming false live/resilience claims.