V3 Day-0 Patch Pipeline Runbook

runbook-set: v3-day0-patch-pipeline.v1

Owner: V3 Release Captain (patch captain for the launch window), joint with the platform-release engineer per store

Companion gates: V3/DAY0_PATCH_HEADROOM.md (size budgets the patch must fit), V3/COOK_SIZE_REPORTS_CI.md (size regression checks), V3/SERVICE_WORKER_CACHE_VALIDATION.md (web cache invalidation), V3/launch/store-rejection-contingency.md (if a patch itself is rejected)

DAY0_PATCH_HEADROOM.md proves a day-0 patch fits — 1 GB reserved on desktop/handheld/console, 500 MB on mobile/standalone VR, delivered as a UE Patcher + Iostore delta with no full installer re-cook. This runbook is the other half: how that patch actually ships fast, end to end, per platform, with the server-vs-client decision made before panic sets in.

Standing Rule: Server-Side First#

Most launch-day defects are fixable without touching a store. The BFF, world server, realtime gateway, pxstream relay, web client, and CDN-delivered assets all deploy on our own clock. A client binary patch is the last resort, taken only when the defect lives in shipped native code.

Mitigation decision matrix#

The patch captain makes this call within 30 minutes of defect confirmation and records it (operator.release.day0_patch.decision) with the matrix row cited.

Client Patch Path: Build → QA Smoke → Cert → Rollout#

Stage 1 — Build (all platforms in parallel, target: 3 h)#

1. Branch from the GA release tag; fix lands with mandatory review even under time pressure (two reviewers, one outside the authoring team).
2. Build UE Patcher + Iostore delta against the shipped GA paks for every affected platform — the same artifact class as the validated synthetic patch (day0-localization-audio-texture-hotfix).
3. Size gate, fail-closed: delta must fit the platform headroom from DAY0_PATCH_HEADROOM.md (1 GB desktop/console, 500 MB mobile/standalone VR). The validated synthetic patch sizes (0.38–0.82 GB) show real multi-pak patches fit with margin; a delta that exceeds headroom does not ship — re-scope the patch, never raise the budget mid-incident.
4. No full re-cook, ever, on the day-0 path. If the fix forces a full re-cook, it is by definition not a day-0 patch — escalate to the release captain for a scheduled update instead.

Stage 2 — QA smoke (target: 2 h, parallel per platform)#

The day-0 smoke suite is a fixed list, not improvised:

• patch-application test: delta applies cleanly over a pristine GA install AND over an install that already took a previous delta;
• cold join to the Atrium within the 5 s budget on the patched build;
• reconnect within the 3 s budget;
• 15-minute soak at target FPS on one representative device per platform (the same devices as the FPS validation gates);
• regression check on the specific defect being fixed, plus its nearest neighbors;
• cook-size report diff vs the GA baseline (COOK_SIZE_REPORTS_CI.md).

Any smoke failure stops that platform's submission; platforms pass or fail independently.

Stage 3 — Cert / store submission (the long pole)#

Expected turnaround per platform; the patch captain plans comms around the slowest affected platform:

Consequence the matrix makes explicit: a native-only defect on Quest or PSVR 2 cannot be fully fixed for 3–5 business days. The kill-switch row of the decision matrix is therefore mandatory engineering, not nice-to-have — every launch feature reachable from a console/VR client needs a server-side disable path, because cert latency is the floor on fixing it client-side.

Stage 4 — Staged rollout#

Server-side: canary 5% of shards (30 min bake, error-rate and tick-health watch) → 50% (30 min) → 100%.

Client, per store mechanics:

Bake-stage abort: crash-free-sessions < 99.5% on the patched cohort, or the defect's regression metric not improving, halts promotion and triggers rollback.

Rollback#

• Server: previous deploy retained warm; rollback is a redeploy, < 15 min.
• Client paks: the previous pak set is retained on disk by the patcher; rollback ships as another forward delta that restores prior content — it fits the same reserved headroom (budgets sized for patch-sized deltas, per DAY0_PATCH_HEADROOM.md), so rollback never requires a full re-download either.
• Steam: repoint default branch at the previous build id (minutes).
• Google Play: halt staged rollout; previous version keeps serving the un-rolled-out cohort; full rollback via a new submission of the prior artifact.
• Apple/Meta/Sony: no true store rollback — rollback is the server kill switch plus an expedited corrective submission. This asymmetry is another reason client patches are last resort.
• Audit event: operator.release.day0_patch.rolled_back with the stage, cohort percentage, and trigger metric recorded.

Communications#

• Internal: patch case opened in the Operator Console at decision time; status at each stage transition.
• Public: status page note when the defect is user-visible (incident-communications action SLAs apply: 15 min to first public update); patch notes published at 100% rollout; per-platform "fix is in review with " honesty when cert is the bottleneck.

Fail-Closed Criteria#

This pipeline is GA-blocking when any of the following holds:

• the Sony day-0 patch window is not confirmed in writing by GA-minus-14 days, or the Meta expedite request channel is unverified by GA-minus-7;
• any launch feature reachable from a console/VR client lacks a tested server-side kill switch (drill: flip each switch in staging, verify the client degrades without crashing — evidence drill-kill-switch-matrix-ga);
• the day-0 smoke suite is not runnable as a single command per platform within the 2 h target;
• a dry-run of this pipeline (synthetic defect → patch → smoke → mock submission → staged rollout in staging) has not been executed end to end before GA, with stage timings recorded (evidence drill-day0-pipeline-ga);
• patch deltas are not verified against the DAY0_PATCH_HEADROOM.md budgets in CI for every affected platform.