# V7 World-State Rollback — Dupe-Exploit Recovery for Persistent Realms Status: V7 operations + Nephthys design. Source: fills the planning gap identified in `V1_V7_PLAN_SET_AUDIT_2026-06-12.md` §6.2 (no exploit-recovery / rollback design). Grounded in `V7/V7_ARCHITECTURE.md` (arch§"Nephthys — Persistence and the Replication Layer" — event-sourced + snapshot-checkpointed, arch§"Nàná — Character and Economy Data Model" — double-entry ledger, arch§"Dependency Resolution and Content-Addressed Storage") and `V7/V7_features.md` (features§"Persistence and the Replication Layer", features§"AI Balance Verification", features§"The Economy Firewall and Anti-Fraud"). Owner: Nephthys lead (mechanism), realm-operations lead (runbook), Eunomia/Themis (oversight). Date: 2026-06-12. ## 1. Why This Is a Staple, Not an Edge Case Every persistent RP server eventually faces a duplication or minting exploit — an inventory race, a trade-window dupe, a mod combination that mints currency (the exact case Ixchel's balance verification exists to catch pre-deploy, features§"AI Balance Verification" — but post-deploy exploits will still happen). On FiveM, recovery is a MySQL backup restore: hours of everyone's progress destroyed to undo one exploiter, or the economy silently absorbs the inflation. V7's event-sourced Nephthys (arch§"Nephthys") makes a categorically better answer possible; this document specifies it: point-in-time recovery, **surgical** rollback that punishes the exploit without taxing the innocent, the operational runbook, and the controls on the rollback power itself. ## 2. Nephthys Point-in-Time Recovery (PITR) Mechanics, building directly on the event-sourced + snapshot-checkpointed store: - **Event log retention**: the per-realm append-only event log is retained **90 days** at full granularity, then compacted to snapshot-only history (planning assumption adopted 2026-06-12; storage cost is bounded by §2 arithmetic below). The economy ledger's journal is retained **24 months** (it is the fraud/forensics record — features§"The Economy Firewall"). - **Snapshot cadence**: per-realm aggregate snapshots every **15 minutes** plus on lifecycle events (resource hot-load, realm update, federation treaty change). RPO for full-realm PITR is therefore the event-replay granularity (effectively per-event); the 15-minute snapshot only bounds replay time. - **Recovery to T-minus**: PITR = load latest snapshot ≤ T, replay events to exactly T (event ordering is total per aggregate; cross-aggregate causality is preserved by the log's per-realm sequence). Determinism of the sim core (arch§"Determinism") is what makes replay-to-T well-defined. - **Replay-time target**: ≤10 minutes to materialize a 256-pop realm's state at arbitrary T within retention; ≤30 minutes for a meshed 1000-pop realm (planning assumption adopted 2026-06-12; validated in the recovery drill, §7). - **Forked materialization, not destructive restore**: PITR always materializes to a **shadow projection** first — operators and platform responders inspect the T-minus state (diff vs. live) before any cutover. Cutover is itself an event ("realm state superseded by PITR-to-T, authorized by X"), so the rollback is in the log and is itself auditable and reversible within retention. - **What PITR can never touch**: real-money state. Purchases, entitlements, payouts live in Aje on the platform plane (arch§"The Trust Boundary") and are **out of scope of any realm rollback** — a realm PITR cannot un-buy a cosmetic or claw a payout. Entitlement *grants into the realm* re-apply on rollback via the entitlement reconciliation job (re-grant anything platform records say the player owns). Storage arithmetic (planning assumption adopted 2026-06-12): a busy 256-pop realm generating ~50 events/s at ~300 B/event ≈ 1.3 GB/day ≈ 117 GB per 90-day window before compression; with columnar compression (~5×) ≈ ~25 GB per realm — at object-storage rates (~$0.02/GB-mo) ≈ $0.50/realm-month. PITR retention is cheap; it ships for every realm tier above Solo. ## 3. Surgical Rollback — Compensating Transactions Full-realm PITR punishes everyone for one exploiter. The default tool is surgical reversal: **append compensating events** against only the affected aggregates — never rewriting history, always adding to it (the double-entry discipline of arch§"Nàná" extended to items). ### 3.1 Currency clawback The realm economy is a double-entry ledger, so exploit proceeds are traceable flows. Procedure: identify minting/dupe entries by ledger query (§5 step 3), compute the tainted subgraph (proceeds + downstream transfers, the same flow tracing as the RMT graph — arch§"The Economy Firewall and Anti-Fraud"), then post clawback entries: - direct exploiter balances: debit the full tainted amount (balances may go negative — a frozen-debt state that gates spending until worked off or adjudicated); - downstream innocent recipients (sold a car to the exploiter for duped cash): default policy is **good-faith-recipient keeps the goods, clawback stops at the exploiter unless the counterparty is graph-flagged as colluding** (same-device/never-spent clusters etc.). This default is realm- charter-overridable (Eunomia), but the platform default protects innocents. Planning assumption adopted 2026-06-12. ### 3.2 Item-instance revocation Every item instance is content-addressed + instance-identified in the inventory aggregate (instance IDs are minted server-side; arch§"Nàná" `inventory[]`). A dupe produces distinguishable instances with identical provenance lineage — two instances claiming descent from one faucet event. Revocation: query instances by provenance lineage; keep the earliest-sequenced instance; append `item-instance-revoked` events for the clones wherever they now sit (inventories, property storage, market escrow). Containers/trades referencing a revoked instance resolve via the same compensation rules as §3.1. Vehicles/property are item instances with extra aggregates and revoke identically. ### 3.3 Player-scoped rollback For a single compromised or exploiting character: replay **that character's aggregate** to T-minus (per-character aggregates are independent projections — arch§"Nephthys": "separate aggregates with independent projections") while the realm stays live. Cross-aggregate edges (trades with others) are settled by compensating entries per §3.1/§3.2 rather than by rolling the counterparties. ### 3.4 Item-class rollback For "the gun that should never have existed" (a mis-configured faucet): revoke all instances of the item class minted in window [T1,T2] by faucet event query, plus optional refund-at-sink-price compensation to holders who acquired in good faith. ## 4. Authority Split — Who Can Trigger What | Action | Realm operator | Platform (realm-ops on-call) | | -------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------- | | Economy freeze (own realm) | Yes — one-click, immediate | Yes | | Surgical compensation (§3.1–§3.4) within own realm | Yes, via the Operator Console tooling | Yes | | Full-realm PITR ≤72h back | Yes, with mandatory shadow-diff review | Yes | | Full-realm PITR >72h back | Request only | Platform executes after review | | Rollback affecting >25% of realm's weekly-active characters | Request, or execute + automatic Eunomia/ Themis post-review (§6) | Platform sign-off required for execution >72h | | Anything touching a **federation corridor** (state that traveled to another realm — features§"Membership: Open, Whitelisted, Federated") | No — request only | Platform coordinates both realms under Themis treaty terms | | Anything touching Aje / entitlements / payouts | Never | Aje-side processes only; realm tools cannot reach it (trust boundary) | | Platform-initiated rollback of a realm against the operator's will | — | Only for safety-floor/legal causes, logged + Themis-reviewable | Rationale for the 72h operator line: it bounds the blast radius of a malicious or panicked operator to a window players can comprehend, while covering the overwhelming majority of real exploit-response needs (planning assumption adopted 2026-06-12). Character deletion limits still hold: rollback is not an erasure loophole — Iris consent rules (features§"Nàná — Persistent Characters") are enforced on the resulting projections, and a rollback that would erase a character record beyond limits is blocked. ## 5. The Exploit-Response Runbook Owner: realm operator for realm-scoped events; platform realm-ops on-call joins at severity ≥S2 (S1 = realm-economy nuisance; S2 = realm-economy material; S3 = cross-realm / firewall-adjacent / suspected platform bug). All steps and timestamps are recorded in the incident record (Operator Console). 1. **Freeze the economy** (target: ≤5 minutes from confirmation). One-click realm economy freeze: ledger writes suspended except payroll accrual; markets/trades/banks read-only; players notified in-client with the incident banner. Freezing first stops taint spread and makes the ledger queryable at a fixed frontier. 2. **Preserve and notify.** The event log is already immutable; pin the current snapshot. If the exploit implicates a resource, capture its exact content-addressed version from the lock file (arch§"Dependency Resolution") — and if that resource is distributed to other realms, escalate to S3 and page Sekhmet (the platform can enumerate every realm pinning the artifact — features§"Incident Response"). 3. **Assess blast radius via ledger queries** (target: ≤2 hours for S2). Standard query pack in the Operator Console: minting anomalies (balance delta vs. faucet-event sum), provenance-duplicate item instances, inflation spike vs. the realm-health baseline (features§"The Economy Firewall": inflation is a first-class metric), affected-account list, taint subgraph depth. 4. **Choose the tool**: surgical (§3) if the taint subgraph is enumerable and <10% of weekly-active characters touched; PITR if state corruption is systemic, the exploit window is short and recent, or taint tracing is intractable. Decision is recorded with reasoning. 5. **Patch the hole before unfreezing**: disable/hot-unload the exploited resource (features§"Hot-Reload Into Live Sessions"), or apply the realm rule fix; re-run the Ixchel balance pass on the fixed composition (features§"AI Balance Verification") — re-verification is mandatory for S2+ before reopen. 6. **Execute** the chosen rollback (shadow-diff review first for PITR; dry-run report first for surgical — "this will post N clawbacks totaling X across M characters"). 7. **Operator comms** — templates shipped in the Operator Console (drafted with Kuanyin comms): (a) freeze notice ("what we froze and why, in one paragraph"); (b) resolution notice (what was rolled back, who is affected and how, what is NOT affected — always explicitly "real-money purchases are unaffected"); (c) post-incident summary for the realm's Eunomia forum. Plain language, no exploit-reproduction details. 8. **Re-open** the economy; monitor the realm-health dashboard at elevated sampling for 72h (inflation, sink coverage, wealth-inequality deltas). 9. **Post-incident review** within 7 days: S2+ incidents file the review to the platform (feeds the balance-verification corpus so Crucible learns the exploit class); §6 oversight runs where triggered. Target end-to-end: S2 exploit from confirmation to economy re-open in **≤24 hours** with surgical tooling (planning assumption adopted 2026-06-12; the FiveM baseline this replaces is days of downtime or a destructive wipe). ## 6. Abuse Prevention of the Rollback Power Itself Rollback is a weapon (an operator could erase a rival faction's winnings or "undo" outcomes they disliked). Controls: - **Total auditability**: every freeze, compensation, and PITR is an event in the same log, attributed to the acting operator account, visible in the realm's admin audit log (features§"Whitelisting and Realm Moderation") — and rollback events are **player-visible** in their own affected-history view ("your balance was adjusted by incident #X"). - **Mandatory Eunomia review** for operator-initiated rollbacks that (a) affect >25% of weekly-active characters, (b) reach >24h back, or (c) recur >2× in 30 days: the action executes (incident response can't wait for a vote) but automatically opens a realm-tier Eunomia review with the incident record attached; an adverse community finding escalates to Themis, whose remedies include reversing the rollback (it's an event — reversible within retention) and operator sanctions. Planning assumption adopted 2026-06-12. - **Rate limits**: ≤1 full-realm PITR per 7 days and ≤4 per 90 days at operator authority; beyond that, platform sign-off — chronic rollback is either an unstable realm (a health problem) or abuse (a Themis problem), and both need eyes. - **No selective memory**: surgical tools operate on query-defined sets (taint subgraph, item class, time window) — the console does not offer "roll back this named player because I said so" without an attached incident record; ad-hoc single-target adjustments route through the existing admin-action audit path and are capped in magnitude (≤1% of realm money supply per 30 days without an incident record; planning assumption adopted 2026-06-12). - **Eval gate** (ADVERSARIAL_EVAL_GATES style; owner Nephthys): `rollback` gate — seeded dupe + minting fixtures must be fully reversed by surgical tooling (ledger sums restored to the cent, clone instances revoked, sole survivor instance correct); a seeded *abusive* rollback (targeting non-tainted accounts) must be blocked or flagged for review; PITR must reproduce the exact T-minus state hash (determinism guard, arch§"Testing, Eval Gates, and Golden Replays"). ## 7. Drills Twice yearly per supported realm tier (Dedicated, Meshed): a scripted dupe is executed on a staging realm; the on-call + a volunteer operator run the §5 runbook end-to-end against the clocks (freeze ≤5 min, assess ≤2 h, reopen ≤24 h, PITR materialization ≤10/30 min). Drill results feed the platform incident-response drill gate (features§"Launch Readiness"). First drill is a meshing-beta entry requirement (features§"Region Rollout Strategy" stage 3).