Ori Store Durability, Disaster Recovery, and Compaction

Status: Planning gap-fill per V1_V7_PLAN_SET_AUDIT_2026-06-12.md §6.2. Date: 2026-06-12. Owners: Ori service owner (apps/v6/egbe-ori-service, Rust) — accountable; Data Platform / DBRE on-call — operates backups, replication, drills; SRE incident commander — RTO/RPO during incidents; V1 audit/compliance reviewer — hash-chain and residency verification.

The Ori is V6's only irreplaceable data. World state can be re-cooked, renders re-run, caches rebuilt; a lost biography is a lost life, and the product's core promise is that "a life cannot be silently rewritten" (V6_features.md:541 –546). Durability here is a trust property, not just an availability property.

Grounding: event store is PostgreSQL + pgvector, partitioned by ori_id, residency-tagged via @oshun/data-residency (V6_ARCHITECTURE.md:557–622, 618–622, 1211–1224); projections snapshot every K events (arch:585–589); continuity hashes already exist for cross-platform projection comparison (arch:1258–1261).

1. Objectives#

During any Ori outage the world keeps running: Moirai's co-located BT/HTN fallback (arch:657–660, 759–760) carries behavior, and new would-be events are buffered durably (§3.5). "A cognition outage costs richness, never the world" (arch:424) extends to the Ori: an Ori outage costs permanence lag, never fabricated events — fail loud, never fake.

2. Replication and backup topology#

Per residency zone (us, eu) — zones never replicate across each other:

            ┌────────────────────────── zone (e.g. eu) ──────────────────────────┐
            │                                                                    │
  writes ──▶│  PRIMARY (AZ-a) ──sync──▶ STANDBY-S (AZ-b)   [synchronous_commit   │
            │      │                                        = on, remote_flush]  │
            │      ├──async──▶ STANDBY-D (AZ-c, recovery_min_apply_delay = 4h)   │
            │      │                                                             │
            │      └──WAL archive (pgBackRest, archive_timeout = 60s)            │
            │                  ▼                                                 │
            │   Object storage (MinIO/S3, in-zone, versioned, object-lock):      │
            │     • continuous WAL          • weekly full + daily differential   │
            │     • sealed event segments   • quarterly archival snapshot (7y)   │
            └────────────────────────────────────────────────────────────────────┘

• Synchronous standby (STANDBY-S): the Ori event-append path commits synchronously to the standby. Appends are the product; a few ms of commit latency is the right trade. Read replicas for projections/Clio reads hang off either node. AZ loss ⇒ RPO 0, automated promotion (supervisor with fencing, e.g. Patroni-class) ⇒ RTO ≤ 5 min.
• Delayed replica (STANDBY-D, 4 h apply delay): the defense against logical disasters — a bad migration, an operator error, an exploit that abuses a code path. Application-level append-only does not protect the DB layer from DROP PARTITION; a 4-hour-behind replica does. (Planning assumption adopted 2026-06-12: 4 h balances blast-radius discovery time against replica staleness during recovery.)
• WAL archiving: archive_timeout = 60s bounds the unarchived window ⇒ worst-case RPO 60 s even if the entire zone's compute is lost and only object storage survives. Object storage is versioned with object-lock (compliance mode) on WAL and segments — a compromised DB credential cannot delete history.
• Base backups: weekly full, daily differential (pgBackRest), 35-day PITR window; quarterly fulls retained 7 years (audit posture, arch:1186–1190).
• No cross-zone replication. EU biographies stay in EU (residency artifact: residencyHomeZone: eu for es-ES). Zone-scale durability comes from multi-AZ + object-storage redundancy, not geo-replication.

3. Recovery procedures by failure class#

Sequencing invariant on any promotion: the Moirai cluster and Aye Bridge are quiesced (writes paused) until the promoted node's per-stream max-vector-clock matches or exceeds the fenced primary's last acknowledged positions, preventing split-brain double-appends. Unmergeable duplicates are impossible by construction (event ids are idempotency keys); the merge is the same machinery as cross-shard reconciliation (arch:608–616).

4. Append-only log archival with integrity hashes#

4.1 Sealed segments#

Events older than 90 days and covered by at least one projection snapshot are copied into immutable sealed segments in object storage:

• Segment = one (ori_id, event-range) batch, serialized as zstd-compressed JSONL of the canonical event encoding, without derived data (embeddings excluded — they are recomputable, §5.3).
• Each segment manifest records: ori_id, first/last event id and vector clocks, event count, sha256(segment_bytes), and prev_segment_hash — a per-stream hash chain, so truncating or editing any historical segment breaks every later manifest.
• A daily fleet-level Merkle root over all manifests is written to the V1 audit platform (and is itself Isis-signed, matching the provenance posture of arch:1294–1298). Tamper-evidence is therefore external to the database.
• Verification: weekly re-hash of a random 1% of segments; quarterly full re-hash; any mismatch is a sev-1 and blocks release.

4.2 Hot-store read path is unchanged#

Postgres keeps an ori_event_segment manifest table; the event-read API serves hot tail ∪ segments transparently. Reads past the hot tail (Book of the Ori generation, audits, projection rebuilds) fault segments in from object storage. Truncation of a fully-archived, hash-verified partition range is the only physical removal ever performed, and it removes nothing logically: the log is the union of segments + tail. Verify-before-truncate is enforced in code and in the drill (§7).

4.3 Database-level append-only enforcement#

• No UPDATE/DELETE privileges on event tables for any role, including the migration role; a trigger raises on any attempt (defense in depth).
• Invariant monitors (page the DBRE): per-stream event count must be monotonic; max(event_id) per partition never decreases; segment manifest count never decreases; daily Merkle root must extend, never replace.

5. Unbounded-growth economics#

5.1 Events/agent/day, derived from the tier cadences (V6_DEPENDENCIES.md:400–437)#

Planning assumptions adopted 2026-06-12, to be re-baselined against staging telemetry:

Active-household agent (8-agent roster, player online 2 h/day):

Wild / Atropos-only agent: ≈ 5 events/game-day ⇒ ≈ 5/day.

5.2 Bytes per event#

Envelope (ids, UTC ts, vector clock ≈ 4 contexts, attribution, provenance ref) ≈ 350 B; mean payload ≈ 1.15 KB (text-bearing events dominate) ⇒ 1.5 KB raw. Memory-class events (MemoryFormed, Reflected; ~66% of an active agent's events) also carry a pgvector embedding: 1,024-dim float32 = 4 KB (embedding model per V1 Iris posture, dep:389–398; planning assumption on dimension). Mean hot footprint ≈ 1.5 KB + 0.66 × 4 KB ≈ 4.2 KB/event.

5.3 Per-agent storage and cost#

5.4 Fleet projection (year-1 planning assumption: 250k household agents ≈ 30–40k active households × 8; 1.5M wild/Commons agents incl. the ≥150 GA seeds growing via the foundry, arch:909–916)#

Conclusions. (1) Storage is not the cost problem — cognition is (see cost-contingency-plan.md); storage is the trust problem and is cheap to do right. (2) The hot tier is dominated by the 90-day tail and the vector index, and archival keeps per-agent hot footprint flat — growth scales with agent population, not with biography depth, which is the property an unbounded append-only design needs. (3) Excluding embeddings from archives (recomputable derived data) is the single biggest archive-size lever (4 KB → 0 of the per-memory-event archive cost).

6. Compaction that preserves "never silently rewritten"#

Hard rule: compaction never deletes, rewrites, merges, or re-encodes a stored event. Every mechanism below is additive or copy-then-verified-move.

1. Projection snapshots every K = 500 events or 30 days per ori_id, whichever first (planning assumption adopted 2026-06-12; arch:585–589 already specifies snapshot-every-K). Snapshots are immutable, versioned by projection-code version, and rebuildable from the log — they are an optimization, never a source of truth.
2. Sealed-segment archival (§4): the cold log moves; it does not shrink.
3. Clio summaries as the queryable hot layer. Chronicle beats, arc threads, and Book chapters (arch:872–902) are the product's actual hot read path over old history. They are projections — Clio "never invents events; its narration is always a read over the authoritative log" (arch:875–876) — so serving summaries hot while raw events go cold changes economics, not truth.
4. Embedding lifecycle. Embeddings are derived data. Memories whose retrieval salience has decayed below threshold for 180 days are evicted from the hot pgvector index; re-embedding on demand from archived text costs one embedding call. Salience reweighting (forgiveness, arch:577–581) and eviction touch the index, never the event.
5. What we will never build: last-writer-wins rollups, destructive "biography summarization" that replaces events, or per-event TTLs. PII handling stays compatible: Ori events reference the steward by opaque ref; account-erasure obligations are met by tombstoning the account↔ref mapping and purging steward-side stores, not by editing biographies (planning assumption adopted 2026-06-12; player-character subjects in V7 need stronger machinery — see ori-schema-evolution.md §4).

7. Restore drill — procedure and pass criteria#

Cadence: quarterly, alternating zones; one annual drill is unannounced. Conductor: DBRE on-call; observer/signoff: Ori service owner + audit reviewer. Results recorded in a proposed release artifact V6/release/ori-dr-readiness.v6release.json behind a proposed verifier verify:v6 ori-dr-readiness (new files; existing gates untouched).

Procedure:

1. Provision a clean cluster in the drill account from the latest weekly full
   • differentials (no access to the live primary).
2. PITR-replay archived WAL to a randomly chosen target time T within the last 24 h; record elapsed wall time and measured WAL gap.
3. Run the integrity verifier: hash-chain check across all sealed segments for a 1% ori_id sample plus the 100 highest-event-count streams; verify the daily Merkle roots extend correctly.
4. Rebuild projections for the sampled streams; compare projection continuity hashes against the live primary's (the mechanism of arch:1258–1261) for streams quiesced at T.
5. Functional probes against the restored instance: append one synthetic event and read it back; mint one passport for a synthetic agent (arch:599–606); run one Clio Chronicle read.
6. Failure-class 3.2 mini-drill annually: simulate a destructive migration, recover via STANDBY-D fast-forward.

Pass criteria (all required):

A failed drill blocks the next V6 release until re-run green, on the same fail-closed footing as the platform-cert and cost gates (arch:512–517, dep:431–437).