V5 Incident Response & On-Call

runbook-set: v5-launch-incident-response.v1

Status: Draft for review — fills audit gap V1_V7_PLAN_SET_AUDIT_2026-06-12 §6.2. Format and operational bar match V3/runbooks/incident-response-v3-ga.md (every runbook drilled before launch, named incident-commander role, audit event written). Drills are scheduled, not yet executed — the drill window is 2027-01-11 → 2027-02-12, ahead of the 2027-03-01 launch; this document does not claim green drills that have not happened.

Owners: Live-Ops SRE Lead (rotation + tooling), V5 Online Services Lead (service runbooks), Trust & Safety Lead (anti-cheat / workshop / economy runbooks), Live-Ops Producer (status page + comms), Postmortem Czar (process).

On-Call Rotation Structure#

Four rotations, each primary + secondary, one-week shifts, handoff Tuesdays 17:00 UTC with a written handoff note:

Coverage: follow-the-sun across us-east, eu-west, ap-northeast hub offices; each rotation has at least one responder awake in-region at all times during launch month, dropping to two-region coverage at steady state. Paging: PagerDuty-equivalent on the Grafana/Prometheus alert stack (V5_DEPENDENCIES.md §15). Ack SLA is per-severity (below); a missed ack auto-escalates to secondary after 5 minutes, then to the pod lead.

Launch month (2027-03): rotations double-staffed for weeks 1–2; the day-of-launch live-service event (features§"Launch Readiness") runs with a named launch commander and a 24-hour war room for the first 72 hours.

Severity Matrix#

Single-player is offline-deterministic by design (features§"V5 Product Promise"), so "the game is unplayable" SEV-1s are online-scope by construction; an incident that breaks the offline path is automatically SEV-1 because it violates the product promise.

Runbooks#

incident: matchmaking-outage#

• Owner: online-services incident commander
• Severity: SEV-2; SEV-1 if all regions or >50% of online CCU affected
• Trigger: ticket error rate >5% for 5 min, or TTM p95 >180 s for 10 min, or match-formed rate drops >60% vs 1-hour baseline.
• First action SLA: acknowledge within 5 minutes; confirm the client retry state (never a hang, features§"Online Services") is rendering within 10 minutes.
• Steps: identify scope (per-cell? per-region? per-provider — PSN/XBL/SDR vs GameLift); fail the affected pool to the healthy provider set; widen rating bands via live config if queue volume is the cause; shed ranked queues before unranked; publish status note.
• Rollback: restore normal pool routing only after TTM p95 <60 s for 15 min.
• Escalation: game-servers on-call (if GameLift placement), then Online Services Lead.
• Audit event: v5.incident.matchmaking_outage.drilled
• Drill: scheduled 2027-01-13 (window 1), evidence id drill-v5-mm-outage

incident: host-migration-cascade-failure#

• Owner: game-servers incident commander
• Severity: SEV-2; SEV-1 if migrations corrupt saves
• Trigger: listen-server migration success <99% over 15 min, or migration time p95 >2 s (the specced bar, features§"Network Quality, LAN, and Server Browser"), or a spike of migration loops (same session migrating >3 times in 5 min).
• First action SLA: acknowledge within 5 minutes; disable new listen-server session creation within 15 minutes if cascading (co-op falls back to dedicated placement).
• Steps: capture migration telemetry for three failing sessions; check whether a client patch changed host-election; route bounded-roster co-op to dedicated servers via live config; verify triple-redundant saves (arch§"Save Corruption Recovery") rolled back cleanly for affected players; publish status note.
• Rollback: re-enable listen hosting after migration success ≥99.5% on a 500- session canary (load-test scenario LT-4 rerun against staging).
• Escalation: Netcode Lead; Save/persistence owner if any corruption confirmed.
• Audit event: v5.incident.host_migration_cascade.drilled
• Drill: scheduled 2027-01-14 (window 1), evidence id drill-v5-hostmig

incident: anti-cheat-false-positive-wave#

• Owner: trust-safety-economy incident commander
• Severity: SEV-1 (players wrongly punished = trust-destroying)
• Trigger: flag rate >5x the 7-day baseline within 1 h, or appeal-intake rate

  10x baseline, or any flag wave correlating with a client/EAC/ML-model deploy in the prior 24 h.

• First action SLA: acknowledge within 2 minutes; freeze the three-strike pipeline (no new shadow-mutes/bans issue) within 10 minutes — flags keep recording, enforcement pauses.
• Steps: identify the offending signal or model version (the ML classifier routes to human review, never auto-bans, features§"Anti-Cheat" — verify that path held); roll back the signal/model config; bulk-revert enforcement actions issued in the wave window; restore standing and forfeited ranked points (the appeals contract, features§"Cross-Platform Social, Reporting, and Bans"); proactive in-client notice + status note to affected players — do not make them appeal individually.
• Rollback: re-enable enforcement only after 24 h of flag rate within 1.5x baseline on the corrected config.
• Escalation: Anti-Cheat ML owner, Trust & Safety Lead, Comms.
• Audit event: v5.incident.anticheat_false_positive_wave.drilled
• Drill: scheduled 2027-01-20 (window 2), evidence id drill-v5-ac-fp

incident: economy-dupe-exploit#

• Owner: trust-safety-economy incident commander
• Severity: SEV-1 (persistent-economy integrity)
• Trigger: per-cell currency-conservation reconciliation breach >0.5% daily drift (the invariant in V5/docs/planning/economy-sink-faucet-analysis.md §4), or wallet-growth anomaly alarm, or a credible exploit report with reproduction.
• First action SLA: acknowledge within 2 minutes; engage circuit breakers within 15 minutes — per-account daily faucet caps tighten to 1x median, and the affected market/board halts (economy doc §5).
• Steps: confirm the dupe vector (save-reconcile race, market round-trip, mission-reward replay); hotfix or disable the vector via live config / live-service-calendar service; quantify minted currency from the ledger; decide remediation tier — targeted clawback for exploiters (ledger-traced), no global rollback unless drift >5% of cell money supply; ban-wave through the standard three-strike review, not auto; publish balance-ledger entry (the public ledger is a launch gate, arch§"Launch Readiness") and status note.
• Rollback: lift circuit breakers after 48 h of clean reconciliation.
• Escalation: Economy Designer, Online Services Lead, Comms for the public ledger note.
• Audit event: v5.incident.economy_dupe.drilled
• Drill: scheduled 2027-01-21 (window 2), evidence id drill-v5-econ-dupe

incident: workshop-malware-or-infringement-wave#

• Owner: trust-safety-economy incident commander
• Severity: SEV-2; SEV-1 if a sandbox escape is credible (workshop content is data-only by spec — executable content implies a parser/loader exploit)
• Trigger: credible report that a published workshop item executes code, crashes clients at scale, or a coordinated infringing-upload wave defeats the ML pre-screen (features§"Workshop & Mods").
• First action SLA: acknowledge within 5 minutes; unpublish the named item(s) within 15 minutes (subscribers' pinned versions flag rather than crash, per spec — verify).
• Steps: for malware-class reports treat as security incident — capture the asset, reproduce in an isolated sandbox, identify the loader vulnerability, hotfix the linter/loader, scan the full catalog for the same asset-hash family and unpublish matches; for infringement waves, add hashes to the known-infringing list, batch-unpublish, and process the DMCA queue per V5/docs/planning/ugc-marketplace-compliance.md §2; notify affected subscribers in-client; publish status note.
• Rollback: republish only items individually re-moderated; loader fix ships before workshop re-opens for the affected content type.
• Escalation: Security on-call (malware class), Workshop service owner, Commerce Counsel (infringement at marketplace scale).
• Audit event: v5.incident.workshop_malware.drilled
• Drill: scheduled 2027-01-27 (window 3), evidence id drill-v5-workshop

incident: cert-revocation#

• Owner: game-servers incident commander
• Severity: SEV-1 (Windows players blocked from online play, or TLS trust broken)
• Trigger: the EAC kernel driver's signing certificate is revoked/expired (Windows blocks driver load → all Windows online play fails, features §"Anti-Cheat"), or a service TLS certificate is revoked/expired, or a platform-store signing issue blocks the client.
• First action SLA: acknowledge within 2 minutes; classify which certificate class within 10 minutes.
• Steps: EAC driver class — engage Epic/EAC support line (vendor SLA in contract), flip the live-config flag that lets online-but-unranked play proceed in EAC-degraded mode on server-side signals only (ranked stays closed — the server-side signal layer keeps running regardless, by design); ship re-signed driver via launcher hotfix. Service TLS class — rotate from the standby intermediate (KMS/Secrets Manager, V5_DEPENDENCIES.md §16), redeploy ingress, verify pinning behavior in client. Publish status note for any player-visible window.
• Rollback: restore ranked/EAC-required surfaces after the re-signed driver reaches >95% of the Windows population.
• Escalation: EAC vendor TAM, Platform cert manager, Security on-call.
• Audit event: v5.incident.cert_revocation.drilled
• Drill: scheduled 2027-02-03 (window 4, tabletop + TLS rotation live drill), evidence id drill-v5-cert

incident: region-outage#

• Owner: incident-command (multi-rotation)
• Severity: SEV-1
• Trigger: an AWS region (one of us-east, us-west, eu-west, ap-northeast, ap-southeast) loses GameLift placement, EKS health, or RDS availability; health-check failure across ≥2 services in one region for 5 min.
• First action SLA: acknowledge within 2 minutes; declare regional failover within 15 minutes.
• Steps: mark region unhealthy in matchmaking placement (players re-home to the next-best-ping region — expect ping up to but not over the 80 ms ceiling, features§"Online Services"); verify GameLift session protection drained what it could; scale surviving regions per the LT-8 drill envelope (capacity plan §5); confirm single-player offline degradation is clean for players mid-session in the dead region (local saves reconcile on reconnect); status page regional component red.
• Rollback: re-home traffic after the region is green for 60 min; ramp 10% → 50% → 100% over 2 h.
• Escalation: AWS TAM, Capacity Lead, exec on-call.
• Audit event: v5.incident.region_outage.drilled
• Drill: scheduled 2027-02-04 (window 4, executes LT-8 against staging), evidence id drill-v5-region

incident: login-auth-outage#

• Owner: online-services incident commander
• Severity: SEV-1
• Trigger: auth error rate >2% for 5 min, JWT issuance p99 >2 s, or a platform-linking provider (PSN/XBL/Steam) failing >10% of link flows.
• First action SLA: acknowledge within 2 minutes; verify offline degradation within 10 minutes (players must drop to offline-deterministic play, never a login wall — features§"Online Services" degradation contract).
• Steps: distinguish our-side (service/RDS/Redis) from provider-side; if provider-side, post status and rely on silent-refresh tokens keeping current sessions alive (15-min refresh gives a grace window); if our-side, roll back last auth deploy, fail over Redis session store; throttle login storm on recovery (queue with honest wait time, 250 RPS ceiling per capacity plan §3).
• Rollback: full traffic after 30 min of error rate <0.1%.
• Escalation: platform first-party contacts, Online Services Lead.
• Audit event: v5.incident.login_outage.drilled
• Drill: scheduled 2027-01-28 (window 3), evidence id drill-v5-auth

incident: telemetry-pipeline-backlog#

• Owner: online-services incident commander
• Severity: SEV-3; SEV-2 if anti-cheat or economy monitoring is blinded >1 h
• Trigger: Kafka consumer lag >15 min, ClickHouse insert failures, or edge ingestion 5xx >1%.
• First action SLA: acknowledge within 15 minutes.
• Steps: confirm clients are buffering (500 events / 2 KB per batch) and not crashing on buffer overflow; scale edge consumers; if ClickHouse is the bottleneck, shed lowest-tier event classes first (cosmetic UX events) via schema-tier config, never anti-cheat signals or economy ledger events; drain backlog; verify no gap in the anti-cheat signal stream (if gapped, extend the affected matches' post-match re-validation window).
• Rollback: restore full event tiers after lag <1 min for 30 min.
• Escalation: Data platform owner, Anti-Cheat ML owner if signal gap >1 h.
• Audit event: v5.incident.telemetry_backlog.drilled
• Drill: scheduled 2027-01-15 (window 1, executes LT-6), evidence id drill-v5-telemetry

incident: marketplace-payment-provider-outage#

• Owner: trust-safety-economy incident commander
• Severity: SEV-2; SEV-3 outside peak hours
• Trigger: /v5/workshop/marketplace/purchase provider error rate >5% for 10 min, or entitlement-grant failures after successful charge (money taken, no item — auto-SEV-2 at any volume), or payout-batch failure.
• First action SLA: acknowledge within 5 minutes; if charge-without- entitlement, pause purchases within 10 minutes.
• Steps: reconcile charge-vs-entitlement from the purchase ledger; auto-grant missing entitlements or auto-refund within the 48 h refund window (year1_modder_marketplace_manifest.json); for payout failures, freeze the batch, dual-control reconcile against the revenue-share ledger (creator 70% / platform 20% / fund 5% / reserves 5%), re-run; status note on the creator hub if payouts slip past the 30-day cadence.
• Rollback: resume purchases after 30 min of provider green + a clean synthetic purchase per region.
• Escalation: payment-provider TAM, Finance operations, Commerce Counsel if payouts slip >5 business days.
• Audit event: v5.incident.marketplace_payments.drilled
• Drill: scheduled 2027-02-10 (window 4), evidence id drill-v5-payments

Status-Page Policy#

• Public status page with per-component rows: login, matchmaking (per cell group), Bureau HQ hub, workshop, marketplace, leaderboards, replays, companion app, per-region health.
• Posting bar: any SEV-1 within 15 min, SEV-2 within 30 min, player-visible SEV-3 within 2 h. Updates per the severity matrix cadence until resolved; every incident closes with a resolution note.
• Language: plain, no blame, no speculation about cause until confirmed; localized into the 12 voiced launch languages within 2 h for SEV-1 (text templates pre-translated for the top-10 incident classes above).
• The in-client retry/degradation states (spec-mandated: never a hang) deep- link to the status page.
• Owner: Live-Ops Producer; the IC supplies facts, the producer publishes.

Postmortem Process#

• Blameless. Required for every SEV-1 and SEV-2, optional for SEV-3 on the pod lead's call.
• Draft within 5 business days; reviewed at the weekly live-ops review; SEV-1 postmortems get a public player-facing summary (matching the transparency posture of the public balance ledger).
• Required sections: timeline (UTC), player impact (CCU-minutes degraded, money/progress affected), detection gap (alert vs first player report), root cause, action items each with a named owner-role and due date.
• Action-item SLA: P0 items 14 days, P1 items 30 days; the Postmortem Czar reviews aging items monthly and escalates slips to pod leads.
• Each postmortem must answer: "would the relevant drill have caught this?" — if yes, why didn't it; if no, update the drill.