Date: 2026-09-15
Task: 13.6
Implementation and execution source: 245b92f03e556f2beb6115652ca995cda1fd52be
Retained receipt: docs/audits/eve-backup-recovery/2026-09-15-05.json
Canonical record digest:
15c193610071a9db85382ed1b8aa8ef98123921d61dd87c54f7b6aa1b55602f1
Decision#
The source-bound Task 13.6 drill passed all 11 required recovery families over the product's production store implementations. It used real local PostgreSQL 16.14, Qdrant 1.19.1, and Linux filesystem substrates. Every family restored the independently counted backup census, produced an equal independently read semantic digest after migration, retained at least one control record, and met its declared RPO and RTO.
The maximum measured RPO was 1 second and the maximum measured RTO was 5 seconds. These are measurements from one isolated drill, not rolling production attainment.
Capability-family results#
| Family | Production substrate | Backup → restored | Retained | Objective RPO | Objective RTO | Measured RPO/RTO | Migration and integrity | Deletion contract |
|---|---|---|---|---|---|---|---|---|
| Conversations | PostgreSQL | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Memory | PostgreSQL | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Vectors | PostgreSQL + Qdrant | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Index metadata | Qdrant | 3 → 3 | 3 | 0 s | 4 h | 0 s/5 s | Pass | Non-subject |
| Workbench | PostgreSQL | 12 → 12 | 12 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Ledger | PostgreSQL | 54 → 54 | 54 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Schedules | PostgreSQL | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Watchers | PostgreSQL | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Task state | PostgreSQL + filesystem | 4 → 4 | 2 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Audit | PostgreSQL | 2 → 2 | 1 | 25 h | 4 h | 1 s/5 s | Pass | Pass |
| Evidence manifests | Filesystem | 2 → 2 | 2 | 6 h | 4 h | 1 s/5 s | Pass | Non-subject |
PostgreSQL authorities inherit the checked nightly-backup age plus one-hour tolerance, so their conservative objective is 90,000 seconds. Evidence files inherit the six-hour off-box cycle. Qdrant vector data is derived from the PostgreSQL authority, and index metadata is recreated from source configuration; the latter therefore has a zero-second data-loss objective.
Production-store and migration boundary#
The drill created isolated source and clean-target PostgreSQL databases and
Qdrant collections. It seeded the actual conversation, Iris memory,
personalization-vector authority/projection, workbench/ledger, schedule,
PostgreSQL watcher, lifecycle, RUN-002 file, audit, and evidence-manifest store
paths. The PostgreSQL database was backed up with pg_dump custom format and
restored with pg_restore; RUN-002 and evidence files were copied with retained
SHA-256 facts.
The fixture deliberately downgraded conversation, memory, vector, schedule,
watcher-fence, lifecycle, and RUN-002 representations before backup. Startup
used the production hydration/migration seams. Conversation and memory current
formats were read back from PostgreSQL, watcher and Workbench deletion-fence
tables were created, lifecycle/RUN-002 current state was persisted, Workbench
projections and their append-only ledger were replayed through the production
store. The recovered Workbench/ledger census now includes accepted goal and
delivery-plan revisions, their normalized requirement/work/proof graph, every
planning event, authenticated fleet activation facts for both the deletion
subject and retained peer, and populated operator-labor trajectory, interval,
coverage, reclassification, and independent QA/product evaluation facts. Legacy
Qdrant payloads were replaced through the deployed PersonalizationVectorIndex
reconciliation path.
Qdrant is intentionally not treated as an authority or snapshot source. The target collection was proven absent, rebuilt from the restored PostgreSQL vector authority, checked for dimensions/distance/indexed payload fields, queried through the real adapter, and read back record by record before admission.
Semantic digests bind subject identity and recovered content while excluding
only migration-local revision counters and the physical source/target Qdrant
collection name. Focused controls prove that changes to conversation content,
memory content, or vector values still change the digest. The first live
candidate correctly failed when those transport-local fields had not yet been
normalized; no record from that attempt was admitted. The first adjacent-plane
candidate likewise failed before evidence emission when its recovery census used
stale delivery-plan column names; the replacement candidate corrected the
bindings and added a populated all-table integration assertion. Independent
review then found that planning_fleet_operation.actor was still absent from
the subject census, redaction scope, snapshot, and admission fence. The current
source repairs that entire boundary, and this receipt is its first and only live
drill.
Deletion and resurrection protection#
Nine families contain the controlled deletion subject: conversations, memory, vectors, workbench, ledger, schedules, watchers, task state, and audit. In each one, the older backup visibly resurrected the subject before replay. The clean target then reconciled one encrypted create-only recovery receipt, verified four service attestations, replayed the actual signed deletion fanout before admission, removed or privacy-redacted only the deleted subject, retained its peer, and rejected a later write that attempted resurrection. Workbench and ledger proof uses authenticated subject actors, recursively discovered event identities/references, the shared write barrier and durable recreation fence, identity-only schema-governed redaction across planning and operator-labor facts—including immutable fleet-operation actor identity—exact projection replay, and independently read post-replay censuses. Both activation and cancellation writers consult the durable subject fence before idempotency replay while holding the shared side of the global erasure barrier.
The published scope-v3 migration remains byte-immutable. Its explicit transaction first takes the same exclusive Workbench write-plane advisory lock used by subject erasure, writes exact-record privacy approvals, redacts every fleet actor still matching a v2 tombstone, proves that the upgraded set has zero remaining matches, and only then advances the fences to v3 and commits. Its database INSERT trigger rejects deleted fleet actors during mixed-version deployment.
A separate ordered follow-up migration takes that barrier, installs a DEFERRABLE INITIALLY DEFERRED inverse constraint trigger on scope-v3 tombstones, repairs any fleet actor exposed by a predecessor eraser that won the narrow gap between the two history entries, proves zero leaks, drains its constraint events, and commits. The deferred check lets the current eraser redact within one transaction but rejects an older eraser at COMMIT before it can report a false deletion success. Startup readiness requires both fleet triggers and the zero-leak invariant.
The Hetzner deploy path now mounts the complete read-only Prisma directory and
runs pinned Prisma 7.8.0 migrate deploy before any application rollout.
Structural schema diff is no longer an apply authority because it omits custom
backfills, barriers, functions, and triggers. Split database credentials are
URL-encoded inside the config rather than exposed as a password-bearing process
argument; absent or inconsistent _prisma_migrations history fails closed for
operator reconciliation and is never silently baselined. The production-path
regression drives that exact config against dirty v2, queues an old eraser
between the two real migration entries, proves the follow-up repairs it, and
then proves a post-migration old eraser is rejected at COMMIT.
Only index metadata and evidence manifests are non-subject schema/release material; their receipt rows carry explicit non-subject reasons.
Production startup now requires the recovery journal, Redis, and attestation signer and refuses HTTP admission until replay succeeds. The production S3 replica adapter requires an independent HTTPS endpoint, create-only writes, Object Lock compliance retention beyond the backup horizon, and readback. The live local drill uses an in-process independent write-once replica boundary; provider IAM and Object Lock behavior remain focused-test evidence rather than a claimed provisioned service.
Fail-closed controls#
All eight retained controls passed:
- stale per-family RPO is rejected;
- a corrupt PostgreSQL archive is rejected;
- malformed Qdrant dimensions/index metadata are rejected;
- an absent target Qdrant collection is rebuilt and queried before admission;
- a corrupt evidence manifest is rejected by the production schema verifier;
- a modified authenticated deletion receipt is rejected;
- restore without completed deletion replay is rejected; and
- startup without any required replay dependency is rejected.
The direct retained-record verifier independently checks the canonical record digest and every source-file hash against the published source commit. It also mutates deletion proof, family completeness, and RPO in memory and requires each negative control to fail before admitting the unchanged record. Its result was 11/11 families, maximum RPO 1 second, and RTO 5 seconds.
Reused source/static receipts#
No green suite was replayed during evidence assembly. The source candidate already had passing receipts for:
- 17 RUN-002 durable-run contract tests;
- 12 PostgreSQL watcher-store tests;
- 6 backup/recovery evaluator tests;
- 35 focused BFF journal, lifecycle, Ori, and vector tests, followed by the exact 12-test lifecycle and 12-test journal/vector repair selections;
- 36 earlier directly affected real-PostgreSQL Workbench, goal, plan, fleet, and operator-labor tests, plus the final 5/5 adjacent-erasure suite covering populated recovery snapshots, identity-only redaction, late-write refusal, both sides of the global create/erase barrier, the immutable 63-entry Prisma ledger, dirty-v2 application of its final two entries, repair of an eraser queued between them, both fleet triggers, and commit-time predecessor rejection;
- 18 subject-data-map tests and 14 production scanner negative controls;
- Task 13.6 runner/verifier contracts and tool typechecking;
- BFF ratchet typechecking with zero backlog and the production server/reprojection CLI build;
- nine affected project typechecks executed serially, the two deployment-path contract tests, shell syntax, changed-source lint with zero errors, and architecture ratchets;
- Docker Compose validation; and
- 16 disaster-recovery runbook operation tests.
The broader workbench spec typecheck has 25 pre-existing errors outside the changed subject-erasure files. Its exact 40-test subject-erasure contract and scanner are green; this unrelated baseline is not represented as passing.
Limits and remaining work#
- This is an isolated local drill over minimized records, not a production restore, regional failover, or rolling RPO/RTO window.
- PostgreSQL and Qdrant each ran as one local node.
- No provisioned production immutable-object repository or IAM ceremony was exercised; that path remains fail-closed until configured.
- Recovery-key rotation remains an operator ceremony, and keys for restorable receipts must remain escrowed through their backup horizon.
- Task 13.6 still requires an updated hash-bound evidence manifest and independent confirmatory and adversarial review before its checkbox can close.
- Phase 13 and G13 remain open: Task 13.5 still needs its credentialed external channel outage lane, and Task 13.7 owns deployed observability and the supervised game day.