V3 Data-Layer DR and Backup Plan

Plan id: v3-dr-backup-plan.v1

Verifier: pnpm verify:v3 dr-backup (proposed)

Evidence artifacts: V3/operations/dr-backup-release-gate.json, V3/operations/dr-restore-drill-<quarter>.json (proposed)

Owner: Capacity engineering (data on-call), joint with Lilith-Privacy lead for consent/audit classes and residency constraints

The POP-loss path is drilled (PIXEL_STREAMING_POP_CAPACITY_MANAGEMENT.md multi-cloud failover), but compute failover is not data protection: nothing to date sets recovery objectives for the data layer itself. This plan sets RPO/RTO per data class, the backup cadence that achieves them, a quarterly restore drill with pass criteria, and the region-loss runbook.

Residency boundary (binding): backups and replicas of residency-pinned data must stay inside the residency zone signed off in V3/PER_REGION_RULES.md (us-west, eu-de, uk, in, br). The pop-capacity residency drill already establishes that India-to-eu-west-1 fallback requires explicit consent — the same rule applies to data: we accept longer downtime over a residency violation.

RPO/RTO Targets per Data Class#

Why these numbers: Redis session loss is survivable by design — the reconnect gate (RECONNECT_VALIDATION.md) and login path rebuild session state, so 60 s of session-state RPO costs reconnects, not data. The consent ledger is the opposite extreme: a consent decision that exists in a user's memory but not in our ledger is a legal exposure (RECORDING_CONSENT_FLOW.md publication gate depends on it), so committed consent transactions get RPO 0 via synchronous replication. Commerce/holdings sit at RPO 5 min because Stripe is an independent source of truth for payments — a ≤5-min gap is reconcilable from provider records via the existing refund/reversal machinery.

Backup Cadence and Retention#

All backups are encrypted with per-zone KMS keys; backup-repo credentials are write-only from production (no production role can delete or overwrite the object-lock buckets — ransomware/insider containment). Backup success and replication lag are alarmed: a missed nightly base backup or WAL-archive lag

5 min (the RPO) pages the data on-call — an unalarmed backup failure silently converts RPO 5 min into RPO 24 h.

Quarterly Restore Drill#

Owner: data on-call engineer of the week; witness: one engineer not on the data rota; evidence: V3/operations/dr-restore-drill-<quarter>.json, reviewed in the weekly ops review.

Procedure (per zone, rotating so every zone is drilled at least annually; eu-de and us-west every quarter):

1. Pick the drill target time T = now − 24 h ± a random offset (the drill must not always restore to a tidy midnight boundary).
2. Provision an isolated restore instance (no production network routes; separate security group) in the same zone.
3. Postgres PITR: pgbackrest --stanza=v3-world-<zone> --type=time --target="<T>" restore then start and let recovery complete. Record wall-clock from command to accepting-connections.
4. Validation, all required:
   • pg_amcheck clean on restored indexes/heap;
   • row-count deltas vs production-at-T (from the hourly stats snapshot) within 0.1% per audited table;
   • ledger coherence: consent-ledger entries for a sampled set of 100 recorded sessions are complete and ordered (the same invariant the publication gate checks);
   • application smoke: a DSAR export (DSAR_V1_PIPELINE.md category set) builds successfully against the restored instance for a synthetic user.
5. Redis: restore the latest RDB snapshot to an isolated instance; verify the surge-queue and session keyspaces deserialize and TTLs are sane.
6. MinIO: restore 3 sampled objects per bucket class from the replica site, verify checksums against the C2PA/manifest hashes where applicable.
7. Consent immutable export: fetch one daily export from the object-lock bucket, verify its signature chain.
8. Write the drill evidence file: timings per step, validation results, restored-to point, deviations.

Pass criteria (all required, fail-closed):

• Postgres accepting-connections within 45 min and validation complete within 60 min (the RTO) — measured, not estimated;
• achieved restore point within 5 min of T (proves the RPO is real);
• consent-ledger DB restore within 30 min;
• zero validation failures in step 4;
• Redis and MinIO restores complete within their RTOs;
• evidence file written and reviewed within 7 days.

A failed drill is a GA-severity finding: the gate goes red, and the fix plus a re-drill must complete within 30 days.

Region-Loss Runbook#

• Owner: Capacity incident commander; severity SEV-1.
• Scenario: total loss of a primary cloud region hosting a residency zone's data plane (not just a POP — the POP-only case is already drilled).

What fails over:

• Pixel Streaming sessions: per the existing multi-cloud drills — secondary cloud capacity in-zone (e.g., Azure centralindia for ap-south-1), re-establishment ~30 s.
• Edge/web/BFF: multi-region edge stays up; BFF traffic shifts to the surviving region serving non-residency-pinned routes.
• Global (non-pinned) Postgres data: promote the cross-region async replica; accept up to RPO 5 min of loss; reconcile commerce gaps against Stripe records before re-enabling payouts.

What is sacrificed (pre-decided, so nobody improvises under pressure):

• All in-flight sessions in the lost region: resume tokens die with the region; users re-login (surge queue armed first — see V3/launch/launch-surge-plan.md, the re-login stampede is a surge).
• Up to 5 min of world/persistence writes (RPO), and up to 60 s of session state.
• Residency-pinned data does not fail over out of zone. If the in zone is lost, India users see a regional outage page until in-zone restore completes (in-zone secondary site or cloud-secondary in-country region); we do not promote an out-of-country replica without the explicit-consent basis established in the residency drill. Same rule for eu-de, uk, br.
• Free-tier capacity in surviving regions is shed first (85% backpressure ladder) to absorb displaced paid/booked users.

Order of operations: declare SEV-1 and assign commander → arm surge queue → fail over pxstream per capacity drills → promote non-pinned replicas → verify consent-ledger synchronous standby state before accepting new recordings anywhere → begin in-zone restore for pinned data from backups → status-page updates every 30 min → commerce reconciliation before payouts re-enable → post-incident review with measured RPO/RTO vs targets.

Fail-Closed Criteria#

The gate fails closed when any of the following holds:

• any data class above lacks a configured, alarmed backup mechanism matching its cadence row;
• WAL-archive or replication lag alarms are missing or routed to no pager;
• the most recent quarterly restore drill is missing, failed, or older than 100 days;
• backup repositories are writable-or-deletable by production roles, or object-lock is absent on consent/audit exports;
• any backup or replica of residency-pinned data exists outside its residency zone;
• the region-loss runbook has no named commander role or has not had a tabletop walk-through before GA (evidence: drill-region-loss-tabletop-ga).