Runbook — Provider Failover

A configured AI, search, payment, messaging, or media provider is unhealthy and the circuit breaker must trip. Covers the failover decision, the failover execution, the customer-visible degradation, and the audit trail.

Scope#

Failover applies to:

• LLM providers serving assistant/Sophia/Metis (multiple model families per domain, primary + alternate routes).
• Embedding/retrieval providers serving Sophia.
• Speech and avatar providers serving voice mode and avatar mode.
• Image and video generation providers serving Isis.
• Messaging providers (Telegram, WhatsApp, email/SMS) per docs/oshun/messaging-boundary.md.
• Payment providers per the billing adapter.

This runbook handles the failover. The deeper subsystem mitigation lives in the subsystem runbooks: assistant-failure.md, grounding-failure.md, model-workflow-rollback.md, etc.

Severity classification#

Source events: provider.health.degraded, provider.failover.triggered, provider.failover.recovered from release-taxonomy.ts; alert assistant-provider-outage from alerts-manifest.ts.

Detection signals#

• Alerts: assistant-provider-outage, generation-pipeline-failover, model-health-degraded.
• Dashboards: provider-health panels on dashboards-service-health.ts; the customer-KPI conversion-success panel for payments.
• Customer signals: spike in assistant.session.error_shown, media.generation.failed, payment.charge_failed, auth.sms_code_failed.
• Partner/status: provider's own status page (page the reliability lead to ack the external status before tripping).

Failover decision#

1. Confirm the provider is the root cause:
   • Error/latency localized to one provider route.
   • Alternate route healthy (latency, success ratio, capacity).
   • Provider's status page corroborates (where available).
2. Confirm capacity for the alternate:
   • Read dashboards-service-health.ts capacity headroom for the alternate.
   • If headroom < forecasted shifted load × 1.5, throttle inbound first (assistant queue-depth, generation queue priority) before failover.
3. Decide route policy:
   • Full failover: 100% traffic moves to alternate.
   • Partial failover: cohort or workload-class moves (e.g., grounded answers move while non-grounded stays on primary).
   • Workload pin: a specific job class pins to alternate while primary stays for everything else.

Failover execution#

1. Trip the circuit breaker via the release-taxonomy.ts provider hook — the hook emits provider.failover.triggered to admin audit and moves the active route to the alternate per the chosen policy.
2. Verify by sampling 20 in-flight sessions/jobs that they land on the alternate route — trace span provider.route attribute must reflect the alternate's identifier.
3. If the affected provider is a payment provider, follow the billing bridge's idempotency-key contract: in-flight charge attempts must be re-keyed against the alternate's idempotency space before retry.
4. If the affected provider is a messaging provider, suppress any non-essential queued sends across the affected channel until the alternate's send-rate and template-compliance baseline is verified.

Communication cadence#

For payment-provider failover, the partner notification is mandatory and must include the idempotency-key strategy and the cohort estimate.

Workload-specific guidance#

• Assistant LLM: failover changes the per-turn cost and may change refusal-rate baselines. Pin Lilith policy to the certified joint baseline for the alternate before sustained customer traffic; otherwise refusal regressions will trigger persona-policy-violation alerts.
• Sophia retrieval/embedding: re-index drift between providers means saved citations may not round-trip — verify a 20-sample retrieval drill before declaring the failover complete.
• Voice/avatar: alternate provider may have different voice IDs and avatar lineages. The assistant must surface the assistant.voice.identity_changed and assistant.avatar.identity_changed disclosure events so the customer perceives the change.
• Generation (Isis): model lineage changes mean watermark and provenance hashes shift. Use the provenance runbook (provenance-failure.md) if signatures fail post-failover.
• Payments: do not re-issue refunds during a failover — pause refund reconciliation until the alternate completes a settlement window.
• Messaging: WhatsApp template approvals, A2P 10DLC registrations, and email DMARC reputation do not transfer. Verify each compliance gate before resuming send.

Escalation#

Standard escalation. For sev1 payments or login-SMS failover, add the compliance lead in parallel with reliability.

Recovery verification#

• Circuit breaker closed and provider.failover.recovered emitted.
• Primary provider error ratio < SLO for 60 min.
• Alternate-provider load returns to baseline (no carry-over throttle).
• Workload-specific verifications: assistant refusal rate within ±1 sigma; Sophia retrieval round-trip OK; Isis provenance signatures valid; payment idempotency reconciliation complete; messaging compliance re-baselined.

Post-incident#

• Postmortem must include: provider route timeline (open → trip → recover), capacity used on alternate, customer-impact estimate per workload class, and any compliance/regulatory notices triggered.
• Update dashboards-service-health.ts capacity-forecast inputs if the alternate hit > 80% utilization during failover.
• File a cost-and-policy review if the alternate's pricing or refusal baseline shifted production economics.