# Ori Store Durability, Disaster Recovery, and Compaction **Status:** Planning gap-fill per `V1_V7_PLAN_SET_AUDIT_2026-06-12.md` §6.2. **Date:** 2026-06-12. **Owners:** Ori service owner (`apps/v6/egbe-ori-service`, Rust) — accountable; Data Platform / DBRE on-call — operates backups, replication, drills; SRE incident commander — RTO/RPO during incidents; V1 audit/compliance reviewer — hash-chain and residency verification. The Ori is V6's only irreplaceable data. World state can be re-cooked, renders re-run, caches rebuilt; a lost biography is a lost _life_, and the product's core promise is that "a life cannot be silently rewritten" (V6_features.md:541 –546). Durability here is a trust property, not just an availability property. Grounding: event store is PostgreSQL + pgvector, partitioned by `ori_id`, residency-tagged via `@oshun/data-residency` (V6_ARCHITECTURE.md:557–622, 618–622, 1211–1224); projections snapshot every K events (arch:585–589); continuity hashes already exist for cross-platform projection comparison (arch:1258–1261). --- ## 1. Objectives | Objective | Value | Scope | | --------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | **RPO** | **≤ 1 minute** (target ≈ 0 for AZ-level failure) | Any committed Ori event | | **RTO (service)** | **≤ 30 minutes** | Failover-class incidents: instance loss, AZ loss, primary corruption detected early | | RTO (full regional rebuild) | ≤ 6 h fleet-wide; ≤ 90 min for active-household partitions (priority restore) | Region-scale loss; see §3.4 — stated honestly rather than pretending 16 TB restores in 30 min | | Integrity | 0 undetected mutations of archived events | Hash-chained sealed segments, §4 | | Residency | Backups and replicas never leave the account home zone | `us` and `eu` zones per `V6/localization/region-rating-residency.v6loc.json` | During any Ori outage the _world keeps running_: Moirai's co-located BT/HTN fallback (arch:657–660, 759–760) carries behavior, and new would-be events are buffered durably (§3.5). "A cognition outage costs richness, never the world" (arch:424) extends to the Ori: an Ori outage costs _permanence lag_, never fabricated events — fail loud, never fake. --- ## 2. Replication and backup topology Per residency zone (`us`, `eu`) — zones never replicate across each other: ``` ┌────────────────────────── zone (e.g. eu) ──────────────────────────┐ │ │ writes ──▶│ PRIMARY (AZ-a) ──sync──▶ STANDBY-S (AZ-b) [synchronous_commit │ │ │ = on, remote_flush] │ │ ├──async──▶ STANDBY-D (AZ-c, recovery_min_apply_delay = 4h) │ │ │ │ │ └──WAL archive (pgBackRest, archive_timeout = 60s) │ │ ▼ │ │ Object storage (MinIO/S3, in-zone, versioned, object-lock): │ │ • continuous WAL • weekly full + daily differential │ │ • sealed event segments • quarterly archival snapshot (7y) │ └────────────────────────────────────────────────────────────────────┘ ``` - **Synchronous standby (STANDBY-S):** the Ori _event-append_ path commits synchronously to the standby. Appends are the product; a few ms of commit latency is the right trade. Read replicas for projections/Clio reads hang off either node. AZ loss ⇒ RPO 0, automated promotion (supervisor with fencing, e.g. Patroni-class) ⇒ RTO ≤ 5 min. - **Delayed replica (STANDBY-D, 4 h apply delay):** the defense against _logical_ disasters — a bad migration, an operator error, an exploit that abuses a code path. Application-level append-only does not protect the DB layer from `DROP PARTITION`; a 4-hour-behind replica does. (Planning assumption adopted 2026-06-12: 4 h balances blast-radius discovery time against replica staleness during recovery.) - **WAL archiving:** `archive_timeout = 60s` bounds the unarchived window ⇒ **worst-case RPO 60 s** even if the entire zone's compute is lost and only object storage survives. Object storage is versioned with object-lock (compliance mode) on WAL and segments — a compromised DB credential cannot delete history. - **Base backups:** weekly full, daily differential (pgBackRest), 35-day PITR window; quarterly fulls retained 7 years (audit posture, arch:1186–1190). - **No cross-zone replication.** EU biographies stay in EU (residency artifact: `residencyHomeZone: eu` for es-ES). Zone-scale durability comes from multi-AZ + object-storage redundancy, not geo-replication. --- ## 3. Recovery procedures by failure class | # | Failure | Mechanism | RPO | RTO | | --- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | 3.1 | Primary instance/AZ loss | Promote STANDBY-S (automated, fenced) | 0 | ≤ 5 min | | 3.2 | Logical corruption (bad deploy, operator error) | Halt writes; fast-forward STANDBY-D to the last-known-good LSN; promote | ≤ window since corruption (bounded by detection; see invariant monitors §4.3) | ≤ 30 min | | 3.3 | Both replicas lost / zone compute loss | PITR from object storage: latest full + differentials + WAL replay | ≤ 60 s | see 3.4 | | 3.4 | Region-scale restore at fleet size (~16 TB hot, §5) | Parallel pgBackRest restore, **partition-priority order**: (1) active-household `ori_id` partitions (players seen in last 14 days), (2) Commons-resident agents, (3) wild/Atropos agents, (4) frozen (transcended/died) Oris last — they are read-only and Clio can serve their Books from rendered artifacts meanwhile | ≤ 60 s | active partitions ≤ 90 min; fleet ≤ 6 h (planning assumption adopted 2026-06-12: sustained 1 GB/s restore throughput; validated in drills) | | 3.5 | Ori unavailable, world still up | World runs BT/HTN; Moirai/world events that would append to the Ori spill to a durable Redis Streams buffer (already the world-event bus, arch:1216–1217) with 24 h capacity; on recovery the buffer drains through the normal vector-clock merge path — buffered events carry their original timestamps and shard clocks, so ordering is preserved and nothing is fabricated | 0 (buffered) | player-visible: none for behavior; Chronicle/reflection lag until drained | **Sequencing invariant on any promotion:** the Moirai cluster and Aye Bridge are quiesced (writes paused) until the promoted node's per-stream max-vector-clock matches or exceeds the fenced primary's last acknowledged positions, preventing split-brain double-appends. Unmergeable duplicates are impossible by construction (event ids are idempotency keys); the merge is the same machinery as cross-shard reconciliation (arch:608–616). --- ## 4. Append-only log archival with integrity hashes ### 4.1 Sealed segments Events older than **90 days** _and_ covered by at least one projection snapshot are copied into immutable **sealed segments** in object storage: - Segment = one `(ori_id, event-range)` batch, serialized as zstd-compressed JSONL of the canonical event encoding, **without** derived data (embeddings excluded — they are recomputable, §5.3). - Each segment manifest records: `ori_id`, first/last event id and vector clocks, event count, `sha256(segment_bytes)`, and `prev_segment_hash` — a **per-stream hash chain**, so truncating or editing any historical segment breaks every later manifest. - A daily fleet-level **Merkle root** over all manifests is written to the V1 audit platform (and is itself Isis-signed, matching the provenance posture of arch:1294–1298). Tamper-evidence is therefore external to the database. - Verification: weekly re-hash of a random 1% of segments; quarterly full re-hash; any mismatch is a sev-1 and blocks release. ### 4.2 Hot-store read path is unchanged Postgres keeps an `ori_event_segment` manifest table; the event-read API serves `hot tail ∪ segments` transparently. Reads past the hot tail (Book of the Ori generation, audits, projection rebuilds) fault segments in from object storage. Truncation of a fully-archived, hash-verified partition range is the **only** physical removal ever performed, and it removes nothing logically: the log is the union of segments + tail. Verify-before-truncate is enforced in code and in the drill (§7). ### 4.3 Database-level append-only enforcement - No `UPDATE`/`DELETE` privileges on event tables for any role, including the migration role; a trigger raises on any attempt (defense in depth). - Invariant monitors (page the DBRE): per-stream event count must be monotonic; `max(event_id)` per partition never decreases; segment manifest count never decreases; daily Merkle root must extend, never replace. --- ## 5. Unbounded-growth economics ### 5.1 Events/agent/day, derived from the tier cadences (V6_DEPENDENCIES.md:400–437) Planning assumptions adopted 2026-06-12, to be re-baselined against staging telemetry: **Active-household agent** (8-agent roster, player online 2 h/day): | Event class | Derivation | Events/day | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------- | | `MemoryFormed` | ~1 meaningful episode per 4 in-scene minutes across Clotho/Lachesis activity | 25 | | `Reflected` | 2 h online ⇒ ~12 Lachesis ticks at 10-game-min cadence; ~⅔ conclude in a written reflection | 8 | | `RelationshipChanged` | social loop in grounds/Commons | 6 | | Objective lifecycle (`ObjectiveAccepted/Refused`, completions) | a few directives/day | 3 | | `ArcAdvanced` / `SkillLearned` | slow burn | 2 | | Other (`BondChanged`, occasional `Crossroads`) | | 1 | | Offline advance (22 h in Atropos, ~1 game-day) | "a handful of story-beats per game-day" | 5 | | **Total** | | **≈ 50 events/day** | **Wild / Atropos-only agent:** ≈ 5 events/game-day ⇒ **≈ 5/day**. ### 5.2 Bytes per event Envelope (ids, UTC ts, vector clock ≈ 4 contexts, attribution, provenance ref) ≈ 350 B; mean payload ≈ 1.15 KB (text-bearing events dominate) ⇒ **1.5 KB raw**. Memory-class events (`MemoryFormed`, `Reflected`; ~66% of an active agent's events) also carry a pgvector embedding: 1,024-dim float32 = 4 KB (embedding model per V1 Iris posture, dep:389–398; planning assumption on dimension). Mean hot footprint ≈ 1.5 KB + 0.66 × 4 KB ≈ **4.2 KB/event**. ### 5.3 Per-agent storage and cost | Quantity | Active-household agent | Wild agent | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | Raw growth (no archival) | 50 × 4.2 KB ≈ 210 KB/day ≈ **77 MB/yr** (×1.5 PG index/TOAST overhead ⇒ ~115 MB/yr if never compacted) | ≈ 5.7 MB/yr | | Hot steady-state with 90-day archival | 90-day tail ~28 MB + snapshot ~4 MB + hot vector index (top ~2,000 salient memories × 4 KB × ~1.8 HNSW overhead) ~15 MB ⇒ **≈ 47 MB** | ≈ 3 MB | | Archive accrual (events sans embeddings, zstd ≈ 0.45×) | 50 × 1.5 KB × 365 × 0.45 ≈ **11 MB/yr** | ≈ 1.2 MB/yr | | Storage cost/agent-year (hot $0.115/GB-mo × 3 nodes; archive $0.012/GB-mo — planning assumptions) | hot ≈ **$0.19** + archive ≈ $0.002 (yr-1) | ≈ $0.013 | ### 5.4 Fleet projection (year-1 planning assumption: 250k household agents ≈ 30–40k active households × 8; 1.5M wild/Commons agents incl. the ≥150 GA seeds growing via the foundry, arch:909–916) | Tier | Math | Total | | --------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------ | | Hot Postgres | 250k × 47 MB + 1.5M × 3 MB | ≈ **16.3 TB** ⇒ ≈ $1,900/mo single-node; ≈ $5,700/mo across primary + 2 standbys ⇒ ≈ **$68k/yr** | | Archive growth | 250k × 11 MB + 1.5M × 1.2 MB | ≈ **4.5 TB/yr**, ≈ $55/mo at end of yr-1, growing ~$55/mo/yr — negligible | | Backups (35-day PITR + quarterly fulls) | ~1.3× hot in object storage | ≈ 21 TB ⇒ ≈ $250/mo | **Conclusions.** (1) Storage is _not_ the cost problem — cognition is (see `cost-contingency-plan.md`); storage is the **trust** problem and is cheap to do right. (2) The hot tier is dominated by the 90-day tail and the vector index, and archival keeps per-agent hot footprint **flat** — growth scales with agent _population_, not with biography _depth_, which is the property an unbounded append-only design needs. (3) Excluding embeddings from archives (recomputable derived data) is the single biggest archive-size lever (4 KB → 0 of the per-memory-event archive cost). --- ## 6. Compaction that preserves "never silently rewritten" Hard rule: **compaction never deletes, rewrites, merges, or re-encodes a stored event.** Every mechanism below is additive or copy-then-verified-move. 1. **Projection snapshots** every K = 500 events or 30 days per `ori_id`, whichever first (planning assumption adopted 2026-06-12; arch:585–589 already specifies snapshot-every-K). Snapshots are immutable, versioned by projection-code version, and rebuildable from the log — they are an _optimization_, never a source of truth. 2. **Sealed-segment archival** (§4): the cold log moves; it does not shrink. 3. **Clio summaries as the queryable hot layer.** Chronicle beats, arc threads, and Book chapters (arch:872–902) are the product's actual hot read path over old history. They are projections — Clio "never invents events; its narration is always a read over the authoritative log" (arch:875–876) — so serving summaries hot while raw events go cold changes economics, not truth. 4. **Embedding lifecycle.** Embeddings are derived data. Memories whose retrieval salience has decayed below threshold for 180 days are evicted from the hot pgvector index; re-embedding on demand from archived text costs one embedding call. Salience reweighting (forgiveness, arch:577–581) and eviction touch the _index_, never the event. 5. **What we will never build:** last-writer-wins rollups, destructive "biography summarization" that replaces events, or per-event TTLs. PII handling stays compatible: Ori events reference the steward by opaque ref; account-erasure obligations are met by tombstoning the account↔ref mapping and purging steward-side stores, not by editing biographies (planning assumption adopted 2026-06-12; player-character subjects in V7 need stronger machinery — see `ori-schema-evolution.md` §4). --- ## 7. Restore drill — procedure and pass criteria **Cadence:** quarterly, alternating zones; one annual drill is unannounced. **Conductor:** DBRE on-call; **observer/signoff:** Ori service owner + audit reviewer. Results recorded in a proposed release artifact `V6/release/ori-dr-readiness.v6release.json` behind a proposed verifier `verify:v6 ori-dr-readiness` (new files; existing gates untouched). Procedure: 1. Provision a clean cluster in the drill account from the latest weekly full - differentials (no access to the live primary). 2. PITR-replay archived WAL to a randomly chosen target time T within the last 24 h; record elapsed wall time and measured WAL gap. 3. Run the integrity verifier: hash-chain check across all sealed segments for a 1% `ori_id` sample plus the 100 highest-event-count streams; verify the daily Merkle roots extend correctly. 4. Rebuild projections for the sampled streams; compare projection **continuity hashes** against the live primary's (the mechanism of arch:1258–1261) for streams quiesced at T. 5. Functional probes against the restored instance: append one synthetic event and read it back; mint one passport for a synthetic agent (arch:599–606); run one Clio Chronicle read. 6. Failure-class 3.2 mini-drill annually: simulate a destructive migration, recover via STANDBY-D fast-forward. Pass criteria (all required): | Check | Bar | | --------------------------------------------------------- | --------------------------------------------------------------------------- | | Measured RPO (WAL gap at T) | ≤ 60 s | | Failover-class RTO (3.1 simulated promotion) | ≤ 5 min | | Priority-restore RTO (active-household sample partitions) | ≤ 90 min | | Hash-chain verification | 0 breaks | | Projection continuity hashes vs live | 100% match on quiesced sample | | Functional probes (append, passport, Chronicle) | all pass | | Drill runbook drift | 0 undocumented manual steps (any improvisation ⇒ runbook PR before signoff) | A failed drill blocks the next V6 release until re-run green, on the same fail-closed footing as the platform-cert and cost gates (arch:512–517, dep:431–437).