# V5 Incident Response & On-Call runbook-set: `v5-launch-incident-response.v1` Status: Draft for review — fills audit gap V1_V7_PLAN_SET_AUDIT_2026-06-12 §6.2. Format and operational bar match `V3/runbooks/incident-response-v3-ga.md` (every runbook drilled before launch, named incident-commander role, audit event written). Drills are **scheduled, not yet executed** — the drill window is 2027-01-11 → 2027-02-12, ahead of the 2027-03-01 launch; this document does not claim green drills that have not happened. Owners: Live-Ops SRE Lead (rotation + tooling), V5 Online Services Lead (service runbooks), Trust & Safety Lead (anti-cheat / workshop / economy runbooks), Live-Ops Producer (status page + comms), Postmortem Czar (process). --- ## On-Call Rotation Structure Four rotations, each **primary + secondary**, one-week shifts, handoff Tuesdays 17:00 UTC with a written handoff note: | Rotation | Scope | Staffed from | | ------------------------- | --------------------------------------------------------------------------- | ------------------------------------- | | **online-services** | The 16 NestJS services, EKS, RDS/Redis, Kafka/ClickHouse pipeline | Online services + data platform pods | | **game-servers** | GameLift fleets, dedicated/listen session health, netcode envelope, relays | Netcode + infra pods | | **trust-safety-economy** | Anti-cheat, workshop moderation surges, marketplace, world-economy ledger | T&S engineering + economy design pods | | **incident-command** | IC duty for SEV-1/SEV-2; comms; status page | Senior engineers + producers (trained IC pool, min 8 people) | Coverage: follow-the-sun across us-east, eu-west, ap-northeast hub offices; each rotation has at least one responder awake in-region at all times during launch month, dropping to two-region coverage at steady state. Paging: PagerDuty-equivalent on the Grafana/Prometheus alert stack (`V5_DEPENDENCIES.md` §15). Ack SLA is per-severity (below); a missed ack auto-escalates to secondary after 5 minutes, then to the pod lead. Launch month (2027-03): rotations double-staffed for weeks 1–2; the day-of-launch live-service event (features§"Launch Readiness") runs with a named launch commander and a 24-hour war room for the first 72 hours. ## Severity Matrix | Severity | Definition | Ack SLA | Mitigation target | Comms | | --------- | ------------------------------------------------------------------------------------------------ | ------- | ----------------- | ------------------------------------------------------ | | **SEV-1** | Players broadly cannot play online, lose progress, lose money, or a security/integrity breach is active | 2 min | 60 min | Status page within 15 min, updates every 30 min; IC assigned; exec page | | **SEV-2** | A major mode or service degraded for many players; single region down; SLO hard-breach | 5 min | 4 h | Status page within 30 min, hourly updates; IC assigned | | **SEV-3** | Partial degradation with workaround; error-budget fast burn; single-feature outage | 15 min | 1 business day | Status page component note if player-visible | | **SEV-4** | Cosmetic/minor; no player-visible impact | next business day | backlog | None | Single-player is offline-deterministic by design (features§"V5 Product Promise"), so "the game is unplayable" SEV-1s are online-scope by construction; an incident that breaks the **offline** path is automatically SEV-1 because it violates the product promise. --- ## Runbooks ## incident: matchmaking-outage - Owner: online-services incident commander - Severity: SEV-2; SEV-1 if all regions or >50% of online CCU affected - Trigger: ticket error rate >5% for 5 min, or TTM p95 >180 s for 10 min, or match-formed rate drops >60% vs 1-hour baseline. - First action SLA: acknowledge within 5 minutes; confirm the client retry state (never a hang, features§"Online Services") is rendering within 10 minutes. - Steps: identify scope (per-cell? per-region? per-provider — PSN/XBL/SDR vs GameLift); fail the affected pool to the healthy provider set; widen rating bands via live config if queue volume is the cause; shed ranked queues before unranked; publish status note. - Rollback: restore normal pool routing only after TTM p95 <60 s for 15 min. - Escalation: game-servers on-call (if GameLift placement), then Online Services Lead. - Audit event: `v5.incident.matchmaking_outage.drilled` - Drill: scheduled 2027-01-13 (window 1), evidence id `drill-v5-mm-outage` ## incident: host-migration-cascade-failure - Owner: game-servers incident commander - Severity: SEV-2; SEV-1 if migrations corrupt saves - Trigger: listen-server migration success <99% over 15 min, or migration time p95 >2 s (the specced bar, features§"Network Quality, LAN, and Server Browser"), or a spike of migration loops (same session migrating >3 times in 5 min). - First action SLA: acknowledge within 5 minutes; disable new listen-server session creation within 15 minutes if cascading (co-op falls back to dedicated placement). - Steps: capture migration telemetry for three failing sessions; check whether a client patch changed host-election; route bounded-roster co-op to dedicated servers via live config; verify triple-redundant saves (arch§"Save Corruption Recovery") rolled back cleanly for affected players; publish status note. - Rollback: re-enable listen hosting after migration success ≥99.5% on a 500- session canary (load-test scenario LT-4 rerun against staging). - Escalation: Netcode Lead; Save/persistence owner if any corruption confirmed. - Audit event: `v5.incident.host_migration_cascade.drilled` - Drill: scheduled 2027-01-14 (window 1), evidence id `drill-v5-hostmig` ## incident: anti-cheat-false-positive-wave - Owner: trust-safety-economy incident commander - Severity: SEV-1 (players wrongly punished = trust-destroying) - Trigger: flag rate >5x the 7-day baseline within 1 h, or appeal-intake rate >10x baseline, or any flag wave correlating with a client/EAC/ML-model deploy in the prior 24 h. - First action SLA: acknowledge within 2 minutes; **freeze the three-strike pipeline** (no new shadow-mutes/bans issue) within 10 minutes — flags keep recording, enforcement pauses. - Steps: identify the offending signal or model version (the ML classifier routes to human review, never auto-bans, features§"Anti-Cheat" — verify that path held); roll back the signal/model config; bulk-revert enforcement actions issued in the wave window; restore standing and forfeited ranked points (the appeals contract, features§"Cross-Platform Social, Reporting, and Bans"); proactive in-client notice + status note to affected players — do not make them appeal individually. - Rollback: re-enable enforcement only after 24 h of flag rate within 1.5x baseline on the corrected config. - Escalation: Anti-Cheat ML owner, Trust & Safety Lead, Comms. - Audit event: `v5.incident.anticheat_false_positive_wave.drilled` - Drill: scheduled 2027-01-20 (window 2), evidence id `drill-v5-ac-fp` ## incident: economy-dupe-exploit - Owner: trust-safety-economy incident commander - Severity: SEV-1 (persistent-economy integrity) - Trigger: per-cell currency-conservation reconciliation breach >0.5% daily drift (the invariant in `V5/docs/planning/economy-sink-faucet-analysis.md` §4), or wallet-growth anomaly alarm, or a credible exploit report with reproduction. - First action SLA: acknowledge within 2 minutes; engage circuit breakers within 15 minutes — per-account daily faucet caps tighten to 1x median, and the affected market/board halts (economy doc §5). - Steps: confirm the dupe vector (save-reconcile race, market round-trip, mission-reward replay); hotfix or disable the vector via live config / live-service-calendar service; quantify minted currency from the ledger; decide remediation tier — targeted clawback for exploiters (ledger-traced), no global rollback unless drift >5% of cell money supply; ban-wave through the standard three-strike review, not auto; publish balance-ledger entry (the public ledger is a launch gate, arch§"Launch Readiness") and status note. - Rollback: lift circuit breakers after 48 h of clean reconciliation. - Escalation: Economy Designer, Online Services Lead, Comms for the public ledger note. - Audit event: `v5.incident.economy_dupe.drilled` - Drill: scheduled 2027-01-21 (window 2), evidence id `drill-v5-econ-dupe` ## incident: workshop-malware-or-infringement-wave - Owner: trust-safety-economy incident commander - Severity: SEV-2; SEV-1 if a sandbox escape is credible (workshop content is data-only by spec — executable content implies a parser/loader exploit) - Trigger: credible report that a published workshop item executes code, crashes clients at scale, or a coordinated infringing-upload wave defeats the ML pre-screen (features§"Workshop & Mods"). - First action SLA: acknowledge within 5 minutes; unpublish the named item(s) within 15 minutes (subscribers' pinned versions flag rather than crash, per spec — verify). - Steps: for malware-class reports treat as security incident — capture the asset, reproduce in an isolated sandbox, identify the loader vulnerability, hotfix the linter/loader, scan the full catalog for the same asset-hash family and unpublish matches; for infringement waves, add hashes to the known-infringing list, batch-unpublish, and process the DMCA queue per `V5/docs/planning/ugc-marketplace-compliance.md` §2; notify affected subscribers in-client; publish status note. - Rollback: republish only items individually re-moderated; loader fix ships before workshop re-opens for the affected content type. - Escalation: Security on-call (malware class), Workshop service owner, Commerce Counsel (infringement at marketplace scale). - Audit event: `v5.incident.workshop_malware.drilled` - Drill: scheduled 2027-01-27 (window 3), evidence id `drill-v5-workshop` ## incident: cert-revocation - Owner: game-servers incident commander - Severity: SEV-1 (Windows players blocked from online play, or TLS trust broken) - Trigger: the EAC kernel driver's signing certificate is revoked/expired (Windows blocks driver load → all Windows online play fails, features §"Anti-Cheat"), or a service TLS certificate is revoked/expired, or a platform-store signing issue blocks the client. - First action SLA: acknowledge within 2 minutes; classify which certificate class within 10 minutes. - Steps: **EAC driver class** — engage Epic/EAC support line (vendor SLA in contract), flip the live-config flag that lets online-but-unranked play proceed in EAC-degraded mode on server-side signals only (ranked stays closed — the server-side signal layer keeps running regardless, by design); ship re-signed driver via launcher hotfix. **Service TLS class** — rotate from the standby intermediate (KMS/Secrets Manager, `V5_DEPENDENCIES.md` §16), redeploy ingress, verify pinning behavior in client. Publish status note for any player-visible window. - Rollback: restore ranked/EAC-required surfaces after the re-signed driver reaches >95% of the Windows population. - Escalation: EAC vendor TAM, Platform cert manager, Security on-call. - Audit event: `v5.incident.cert_revocation.drilled` - Drill: scheduled 2027-02-03 (window 4, tabletop + TLS rotation live drill), evidence id `drill-v5-cert` ## incident: region-outage - Owner: incident-command (multi-rotation) - Severity: SEV-1 - Trigger: an AWS region (one of us-east, us-west, eu-west, ap-northeast, ap-southeast) loses GameLift placement, EKS health, or RDS availability; health-check failure across ≥2 services in one region for 5 min. - First action SLA: acknowledge within 2 minutes; declare regional failover within 15 minutes. - Steps: mark region unhealthy in matchmaking placement (players re-home to the next-best-ping region — expect ping up to but not over the 80 ms ceiling, features§"Online Services"); verify GameLift session protection drained what it could; scale surviving regions per the LT-8 drill envelope (capacity plan §5); confirm single-player offline degradation is clean for players mid-session in the dead region (local saves reconcile on reconnect); status page regional component red. - Rollback: re-home traffic after the region is green for 60 min; ramp 10% → 50% → 100% over 2 h. - Escalation: AWS TAM, Capacity Lead, exec on-call. - Audit event: `v5.incident.region_outage.drilled` - Drill: scheduled 2027-02-04 (window 4, executes LT-8 against staging), evidence id `drill-v5-region` ## incident: login-auth-outage - Owner: online-services incident commander - Severity: SEV-1 - Trigger: auth error rate >2% for 5 min, JWT issuance p99 >2 s, or a platform-linking provider (PSN/XBL/Steam) failing >10% of link flows. - First action SLA: acknowledge within 2 minutes; verify offline degradation within 10 minutes (players must drop to offline-deterministic play, never a login wall — features§"Online Services" degradation contract). - Steps: distinguish our-side (service/RDS/Redis) from provider-side; if provider-side, post status and rely on silent-refresh tokens keeping current sessions alive (15-min refresh gives a grace window); if our-side, roll back last auth deploy, fail over Redis session store; throttle login storm on recovery (queue with honest wait time, 250 RPS ceiling per capacity plan §3). - Rollback: full traffic after 30 min of error rate <0.1%. - Escalation: platform first-party contacts, Online Services Lead. - Audit event: `v5.incident.login_outage.drilled` - Drill: scheduled 2027-01-28 (window 3), evidence id `drill-v5-auth` ## incident: telemetry-pipeline-backlog - Owner: online-services incident commander - Severity: SEV-3; SEV-2 if anti-cheat or economy monitoring is blinded >1 h - Trigger: Kafka consumer lag >15 min, ClickHouse insert failures, or edge ingestion 5xx >1%. - First action SLA: acknowledge within 15 minutes. - Steps: confirm clients are buffering (500 events / 2 KB per batch) and not crashing on buffer overflow; scale edge consumers; if ClickHouse is the bottleneck, shed lowest-tier event classes first (cosmetic UX events) via schema-tier config, **never** anti-cheat signals or economy ledger events; drain backlog; verify no gap in the anti-cheat signal stream (if gapped, extend the affected matches' post-match re-validation window). - Rollback: restore full event tiers after lag <1 min for 30 min. - Escalation: Data platform owner, Anti-Cheat ML owner if signal gap >1 h. - Audit event: `v5.incident.telemetry_backlog.drilled` - Drill: scheduled 2027-01-15 (window 1, executes LT-6), evidence id `drill-v5-telemetry` ## incident: marketplace-payment-provider-outage - Owner: trust-safety-economy incident commander - Severity: SEV-2; SEV-3 outside peak hours - Trigger: `/v5/workshop/marketplace/purchase` provider error rate >5% for 10 min, or entitlement-grant failures after successful charge (money taken, no item — auto-SEV-2 at any volume), or payout-batch failure. - First action SLA: acknowledge within 5 minutes; if charge-without- entitlement, pause purchases within 10 minutes. - Steps: reconcile charge-vs-entitlement from the purchase ledger; auto-grant missing entitlements or auto-refund within the 48 h refund window (`year1_modder_marketplace_manifest.json`); for payout failures, freeze the batch, dual-control reconcile against the revenue-share ledger (creator 70% / platform 20% / fund 5% / reserves 5%), re-run; status note on the creator hub if payouts slip past the 30-day cadence. - Rollback: resume purchases after 30 min of provider green + a clean synthetic purchase per region. - Escalation: payment-provider TAM, Finance operations, Commerce Counsel if payouts slip >5 business days. - Audit event: `v5.incident.marketplace_payments.drilled` - Drill: scheduled 2027-02-10 (window 4), evidence id `drill-v5-payments` --- ## Status-Page Policy - Public status page with per-component rows: login, matchmaking (per cell group), Bureau HQ hub, workshop, marketplace, leaderboards, replays, companion app, per-region health. - Posting bar: any SEV-1 within 15 min, SEV-2 within 30 min, player-visible SEV-3 within 2 h. Updates per the severity matrix cadence until resolved; every incident closes with a resolution note. - Language: plain, no blame, no speculation about cause until confirmed; localized into the 12 voiced launch languages within 2 h for SEV-1 (text templates pre-translated for the top-10 incident classes above). - The in-client retry/degradation states (spec-mandated: never a hang) deep- link to the status page. - Owner: Live-Ops Producer; the IC supplies facts, the producer publishes. ## Postmortem Process - Blameless. Required for every SEV-1 and SEV-2, optional for SEV-3 on the pod lead's call. - Draft within 5 business days; reviewed at the weekly live-ops review; SEV-1 postmortems get a public player-facing summary (matching the transparency posture of the public balance ledger). - Required sections: timeline (UTC), player impact (CCU-minutes degraded, money/progress affected), detection gap (alert vs first player report), root cause, action items each with a named owner-role and due date. - Action-item SLA: P0 items 14 days, P1 items 30 days; the Postmortem Czar reviews aging items monthly and escalates slips to pod leads. - Each postmortem must answer: "would the relevant drill have caught this?" — if yes, why didn't it; if no, update the drill.