# Regional Failover Runbook

This runbook defines the operational flow for evacuating traffic from an
impaired region to a healthy secondary region. It combines automated guardrails,
explicit approval gates, and validation procedures to guarantee data durability
and minimal downtime.

## Objectives

- Protect customer experience during regional outages.
- Preserve data integrity through controlled promotion of replicas.
- Provide structured communication and audit trails.
- Exercise the plan quarterly via chaos drills with objective SLAs.

## Prerequisites

1. **Observability baselines** – All golden signals published to Grafana
   dashboards `failover/core` and `failover/regional-drill`.
2. **Replica readiness** – Read-only failover clusters pre-provisioned;
   replication latency < 5 seconds.
3. **Service mesh configuration** – Failover weights and locality-aware routing
   defined in Istio `DestinationRule` resources.
4. **Runbook tooling** – `scripts/reliability/run-failover-drill.ts` available
   locally with credentials to reach the control plane.
5. **Stakeholder roster** – Current on-call listings for SRE, platform,
   security, and communications teams.

## Detection and Triage

1. PagerDuty alert `RegionalOutageDetected` fires when 3/5 SLIs breach for
   longer than 3 minutes.
2. Primary on-call triggers this runbook and opens an incident bridge.
3. Confirm impact using dashboards:
   - `latency:p99`
   - `error_rate`
   - `availability`
4. Validate that the secondary region metrics remain within thresholds.

## Failover Execution Steps

1. **Freeze deploys** – `argo app suspend lilith-*` in the primary region.
2. **Promote replicas** – Trigger database promotion via Terraform Cloud run
   `promote-secondary-region`.
3. **Drain traffic** –
   - Update Istio `VirtualService` weights to 0 for the failing region.
   - Confirm Envoy configuration propagation (`istioctl proxy-status`).
4. **Invalidate caches** – Run `scripts/cache/flush-edge.sh --region primary`.
5. **Warm secondary region** – Pre-warm personalization/vector caches using
   `npm run warm:memory -- --region secondary`.
6. **Switch DNS** – Update Route53 failover records; verify TTL expiration.
7. **Smoke testing** – Execute `npm run smoke:failover -- --region secondary`.
8. **Customer comms** – Post status page update and stakeholder summaries.

## Validation Checklist

- [ ] 90% of requests served from secondary region with <10% error rate.
- [ ] Database replication caught up; no divergent writes.
- [ ] Background jobs rebalanced in secondary region.
- [ ] Postmortem template opened in Confluence and populated with timelines.

## Chaos Drill Playbook

1. Schedule quarterly drills with platform, SRE, and product owners.
2. Use `scripts/reliability/run-failover-drill.ts --dry-run` for rehearsal to
   enumerate action items.
3. Execute the drill with
   `scripts/reliability/run-failover-drill.ts --primary us-east-1 --secondary us-west-2`.
4. Require sign-off from:
   - Drill commander (SRE primary on-call).
   - Incident communications lead.
   - Business continuity owner.
5. Capture metrics:
   - Mean time to detect (MTTD)
   - Mean time to route (MTTRoute)
   - Mean time to recover (MTTR)
6. Publish drill summary to `docs/runbooks/drill-reports/YYYY-MM-DD.md` within
   48 hours.

## Circuit Breaker Verification

1. Ensure BFF circuit breakers are configured with sane thresholds
   (`/v1/circuit-breaker/config`).
2. Run automated validation:
   - `npx playwright test tests/reliability.circuit-breaker.spec.ts`
   - `npx playwright test tests/reliability.failover-compliance.spec.ts`
3. During drills, execute
   `scripts/reliability/run-failover-drill.ts --validate-circuit-breakers` to
   confirm circuit state transitions.
4. Observe Prometheus metric `bff_circuit_breaker_state` for each downstream
   service; expect state `half-open` during controlled recovery.
5. Document any manual overrides or anomalies in the incident timeline.

## Post-Failover Activities

1. Verify regional parity after primary region restoration.
2. Reverse DNS and traffic weights gradually while monitoring error budgets.
3. Re-enable CI/CD once stability confirmed for 30 minutes.
4. Conduct post-incident review within 72 hours; track corrective actions.

## References

- `docs/deploy/SERVICE_MESH.md`
- `docs/deploy/OBSERVABILITY_STACK.md`
- `docs/security/INCIDENT_RESPONSE.md`
- `docs/runbooks/FIRE_DRILL_CHECKLIST.md`
