This runbook defines the operational flow for evacuating traffic from an impaired region to a healthy secondary region. It combines automated guardrails, explicit approval gates, and validation procedures to guarantee data durability and minimal downtime.
Objectives#
- Protect customer experience during regional outages.
- Preserve data integrity through controlled promotion of replicas.
- Provide structured communication and audit trails.
- Exercise the plan quarterly via chaos drills with objective SLAs.
Prerequisites#
- Observability baselines – All golden signals published to Grafana
dashboards
failover/coreandfailover/regional-drill. - Replica readiness – Read-only failover clusters pre-provisioned; replication latency < 5 seconds.
- Service mesh configuration – Failover weights and locality-aware routing
defined in Istio
DestinationRuleresources. - Runbook tooling –
scripts/reliability/run-failover-drill.tsavailable locally with credentials to reach the control plane. - Stakeholder roster – Current on-call listings for SRE, platform, security, and communications teams.
Detection and Triage#
- PagerDuty alert
RegionalOutageDetectedfires when 3/5 SLIs breach for longer than 3 minutes. - Primary on-call triggers this runbook and opens an incident bridge.
- Confirm impact using dashboards:
latency:p99error_rateavailability
- Validate that the secondary region metrics remain within thresholds.
Failover Execution Steps#
- Freeze deploys –
argo app suspend lilith-*in the primary region. - Promote replicas – Trigger database promotion via Terraform Cloud run
promote-secondary-region. - Drain traffic –
- Update Istio
VirtualServiceweights to 0 for the failing region. - Confirm Envoy configuration propagation (
istioctl proxy-status).
- Update Istio
- Invalidate caches – Run
scripts/cache/flush-edge.sh --region primary. - Warm secondary region – Pre-warm personalization/vector caches using
npm run warm:memory -- --region secondary. - Switch DNS – Update Route53 failover records; verify TTL expiration.
- Smoke testing – Execute
npm run smoke:failover -- --region secondary. - Customer comms – Post status page update and stakeholder summaries.
Validation Checklist#
- 90% of requests served from secondary region with <10% error rate.
- Database replication caught up; no divergent writes.
- Background jobs rebalanced in secondary region.
- Postmortem template opened in Confluence and populated with timelines.
Chaos Drill Playbook#
- Schedule quarterly drills with platform, SRE, and product owners.
- Use
scripts/reliability/run-failover-drill.ts --dry-runfor rehearsal to enumerate action items. - Execute the drill with
scripts/reliability/run-failover-drill.ts --primary us-east-1 --secondary us-west-2. - Require sign-off from:
- Drill commander (SRE primary on-call).
- Incident communications lead.
- Business continuity owner.
- Capture metrics:
- Mean time to detect (MTTD)
- Mean time to route (MTTRoute)
- Mean time to recover (MTTR)
- Publish drill summary to
docs/runbooks/drill-reports/YYYY-MM-DD.mdwithin 48 hours.
Circuit Breaker Verification#
- Ensure BFF circuit breakers are configured with sane thresholds
(
/v1/circuit-breaker/config). - Run automated validation:
npx playwright test tests/reliability.circuit-breaker.spec.tsnpx playwright test tests/reliability.failover-compliance.spec.ts
- During drills, execute
scripts/reliability/run-failover-drill.ts --validate-circuit-breakersto confirm circuit state transitions. - Observe Prometheus metric
bff_circuit_breaker_statefor each downstream service; expect statehalf-openduring controlled recovery. - Document any manual overrides or anomalies in the incident timeline.
Post-Failover Activities#
- Verify regional parity after primary region restoration.
- Reverse DNS and traffic weights gradually while monitoring error budgets.
- Re-enable CI/CD once stability confirmed for 30 minutes.
- Conduct post-incident review within 72 hours; track corrective actions.
References#
docs/deploy/SERVICE_MESH.mddocs/deploy/OBSERVABILITY_STACK.mddocs/security/INCIDENT_RESPONSE.mddocs/runbooks/FIRE_DRILL_CHECKLIST.md