Eve library

Regional Failover Runbook

Runbook

Revision 1

On this page

This runbook defines the operational flow for evacuating traffic from an impaired region to a healthy secondary region. It combines automated guardrails, explicit approval gates, and validation procedures to guarantee data durability and minimal downtime.

Objectives#

  • Protect customer experience during regional outages.
  • Preserve data integrity through controlled promotion of replicas.
  • Provide structured communication and audit trails.
  • Exercise the plan quarterly via chaos drills with objective SLAs.

Prerequisites#

  1. Observability baselines – All golden signals published to Grafana dashboards failover/core and failover/regional-drill.
  2. Replica readiness – Read-only failover clusters pre-provisioned; replication latency < 5 seconds.
  3. Service mesh configuration – Failover weights and locality-aware routing defined in Istio DestinationRule resources.
  4. Runbook tooling – scripts/reliability/run-failover-drill.ts available locally with credentials to reach the control plane.
  5. Stakeholder roster – Current on-call listings for SRE, platform, security, and communications teams.

Detection and Triage#

  1. PagerDuty alert RegionalOutageDetected fires when 3/5 SLIs breach for longer than 3 minutes.
  2. Primary on-call triggers this runbook and opens an incident bridge.
  3. Confirm impact using dashboards:
    • latency:p99
    • error_rate
    • availability
  4. Validate that the secondary region metrics remain within thresholds.

Failover Execution Steps#

  1. Freeze deploys – argo app suspend lilith-* in the primary region.
  2. Promote replicas – Trigger database promotion via Terraform Cloud run promote-secondary-region.
  3. Drain traffic –
    • Update Istio VirtualService weights to 0 for the failing region.
    • Confirm Envoy configuration propagation (istioctl proxy-status).
  4. Invalidate caches – Run scripts/cache/flush-edge.sh --region primary.
  5. Warm secondary region – Pre-warm personalization/vector caches using npm run warm:memory -- --region secondary.
  6. Switch DNS – Update Route53 failover records; verify TTL expiration.
  7. Smoke testing – Execute npm run smoke:failover -- --region secondary.
  8. Customer comms – Post status page update and stakeholder summaries.

Validation Checklist#

  • 90% of requests served from secondary region with <10% error rate.
  • Database replication caught up; no divergent writes.
  • Background jobs rebalanced in secondary region.
  • Postmortem template opened in Confluence and populated with timelines.

Chaos Drill Playbook#

  1. Schedule quarterly drills with platform, SRE, and product owners.
  2. Use scripts/reliability/run-failover-drill.ts --dry-run for rehearsal to enumerate action items.
  3. Execute the drill with scripts/reliability/run-failover-drill.ts --primary us-east-1 --secondary us-west-2.
  4. Require sign-off from:
    • Drill commander (SRE primary on-call).
    • Incident communications lead.
    • Business continuity owner.
  5. Capture metrics:
    • Mean time to detect (MTTD)
    • Mean time to route (MTTRoute)
    • Mean time to recover (MTTR)
  6. Publish drill summary to docs/runbooks/drill-reports/YYYY-MM-DD.md within 48 hours.

Circuit Breaker Verification#

  1. Ensure BFF circuit breakers are configured with sane thresholds (/v1/circuit-breaker/config).
  2. Run automated validation:
    • npx playwright test tests/reliability.circuit-breaker.spec.ts
    • npx playwright test tests/reliability.failover-compliance.spec.ts
  3. During drills, execute scripts/reliability/run-failover-drill.ts --validate-circuit-breakers to confirm circuit state transitions.
  4. Observe Prometheus metric bff_circuit_breaker_state for each downstream service; expect state half-open during controlled recovery.
  5. Document any manual overrides or anomalies in the incident timeline.

Post-Failover Activities#

  1. Verify regional parity after primary region restoration.
  2. Reverse DNS and traffic weights gradually while monitoring error budgets.
  3. Re-enable CI/CD once stability confirmed for 30 minutes.
  4. Conduct post-incident review within 72 hours; track corrective actions.

References#

  • docs/deploy/SERVICE_MESH.md
  • docs/deploy/OBSERVABILITY_STACK.md
  • docs/security/INCIDENT_RESPONSE.md
  • docs/runbooks/FIRE_DRILL_CHECKLIST.md