# V3 Launch Surge Plan Plan id: `v3-launch-surge-plan.v1` Owner: Capacity / Pixel Streaming incident commander (surge commander for GA day), joint with V3 Release Captain Companion gates: `V3/FLEET_LOAD_VALIDATION.md` (5x fleet envelope), `V3/PIXEL_STREAMING_SESSION_START_SUCCESS.md` and `V3/PIXEL_STREAMING_RTT_VALIDATION.md` (per-POP 5x validation), `V3/PIXEL_STREAMING_ABUSE_POSTURE.md` (85% POP backpressure), `V3/COLD_JOIN_VALIDATION.md` (5 s cold-join budget), `V3/REGION_ROLLOUT_COMPLIANCE.md` (waitlist landing machinery) The load gates validate sustained operation at 5x expected concurrency. They do not validate the GA **moment**: a synchronized signup/login spike driven by marketing, press, and creator drops landing in the same hour. This plan defines the demand assumption above the validated envelope, the virtual waiting room that absorbs the difference, and the rules that keep marketing from stacking demand onto a saturated fleet. ## Demand Model Steady-state assumptions (from the fleet gate): 5,000 concurrent sessions, mean session length 45 minutes, therefore ~110 organic session starts/minute and ~10 signups/minute at steady state. **GA-moment peak assumption: 10x steady-state concurrent demand in hour one — 50,000 users attempting to be in V3 simultaneously — with login/signup arrivals spiking to 30x (≈3,300 auth requests/minute) in the first 15 minutes after the GA announcement.** Decay model: 10x → 4x by hour 6, → 2.5x by hour 24, → settling toward the 5x-validated envelope ceiling during launch week. The validated envelope is 25,000 concurrent (5x fleet gate; per-POP 5x session-start and RTT gates). The structural gap this plan closes: **peak demand (50,000) is 2x validated capacity (25,000).** The waiting room absorbs that difference; nothing else does. We do not attempt to serve 10x — we serve 5x well and queue the rest honestly. Per-POP expected (1x) Pixel Streaming concurrency, from the session-start gate's 5x figures: `us-east-1` 1,200, `us-west-2` 800, `eu-west-1` 800, `ap-northeast-1` 300, `ap-south-1` 250, `ap-southeast-2` 250, `sa-east-1` 250 (total 3,850). Regional admission weights below derive from these. ## Virtual Waiting Room A queue service in front of the BFF auth and launch-resolver routes (`/api/v1/auth/*`, `/api/v3/lilith/launch`), running at the edge/CDN layer so queued users cost the origin nothing. Queue state lives in Redis (separate logical DB from session state); queue tickets are signed JWTs (`type=queue-ticket`, `position`, `issuedAt`, `regionWeightClass`) issued by the same signing infrastructure as session JWTs. ### Entry criteria — who gets queued Queueing activates per-region or globally when ANY of: - global concurrent sessions ≥ 20,000 (80% of the validated 25,000 envelope); - BFF auth or launch-resolver p95 latency > 500 ms over 5 minutes; - session-start success on the user's target POP < 99.5% over 5 minutes; - the user's regional POP is in 85% backpressure (see wiring below); - surge commander manually arms it (always armed for GA hour zero). Bypass lanes (never queued): - active sessions and reconnects carrying a valid resume token — the 3 s reconnect budget (`RECONNECT_VALIDATION.md`) must not be spent in a queue; - booked attendees inside a paid-event pre-warm window (consistent with the concert pre-warm hard cap in the abuse posture); - operator, instructor, and editorial accounts (tenant-scoped grants). ### Token bucket numbers | Bucket | Sustained rate | Burst | Notes | | ---------------------------- | ---------------------- | ----- | ------------------------------------------------------ | | Global admission | 600 admits/min (10/s) | 1,200 | Refills the regional buckets; hard ceiling | | `us-east-1` region | 186/min (31%) | 372 | Regional weights follow per-POP 1x capacity share | | `us-west-2` region | 126/min (21%) | 252 | | | `eu-west-1` region | 126/min (21%) | 252 | | | `ap-northeast-1` region | 48/min (8%) | 96 | | | `ap-south-1` region | 39/min (6.5%) | 78 | | | `ap-southeast-2` region | 39/min (6.5%) | 78 | | | `sa-east-1` region | 39/min (6.5%) | 78 | | | Signup (account creation) | 400/min global | 800 | Separate from session admission; signups can complete | | | | | and park on "you're in line to join a world" | At 600 admits/min, draining a 50,000-deep peak queue takes ~83 minutes if no one leaves — the public wait-time estimate shown to users is computed from the live drain rate, never from a static table. The admission controller treats the bucket rates as **ceilings** and continuously clamps to downstream health: effective admit rate = min(bucket rate, largest rate at which cold-join p95 stays ≤ 5 s AND session-start success stays ≥ 99.5% over the trailing 10 minutes). Health degrades → admits slow before users feel it. ### Queue mechanics - FIFO per region with a single global ordering tiebreaker; no paid queue-jumping at GA. - Admitted users get a 5-minute join window; unexercised admissions return capacity to the bucket. - Queue page shows position, live wait estimate, and offers the Tier-2 web fallback ("join now at lower fidelity") as a no-penalty exit — every user who takes it is load the POPs never see. - Queue ticket TTL 4 hours; refresh is silent; closing the tab keeps your place for 15 minutes (ticket re-presented on return). ## POP Backpressure Wiring The abuse posture already routes free-tier users to fallback web with a banner at 85% POP utilization. The queue extends that into a graduated regional ladder: | POP utilization | Existing behavior | Queue behavior (new) | | --------------- | ---------------------------------- | ------------------------------------------------------------------------- | | < 75% | normal | regional bucket at full rate | | 75–85% | normal | regional bucket halves; queue page starts offering Tier-2 fallback first | | ≥ 85% | free tier → fallback web + banner | regional bucket admits **zero** new tier-3 (pxstream) sessions; admissions continue for native and web-tier joins only | | ≥ 90% | — | regional bucket frozen entirely; region shows queue-paused state; surge commander paged | Backpressure state is read from the same relay signal that drives the 85% banner (`apps/v3/lilith-pxstream-relay`), polled by the admission controller every 15 seconds. The queue must never admit a user into a region whose POP will immediately bounce them to fallback — admit-into-backpressure is a correctness bug, alert-worthy on sight. ## Marketing and Social Coordination Rules Owner: communications lead; enforcement: surge commander has veto, logged in the Operator Console. - **No marketing push of any kind (e-mail blast, push notification, paid social, creator/partner drop) while global queue depth > 5,000 or median wait > 10 minutes.** This is the hard rule the rest are details of. - Every push is pre-sized: expected click-through must fit within 15 minutes of admission capacity (600/min × 15 = 9,000 admits). Blasts above that are split into region-staggered waves at least 60 minutes apart. - Push schedule is gated green/red on the surge dashboard (`grafana://v3/launch/surge`: queue depth, drain rate, admit rate vs ceiling, per-POP utilization, session-start success). Red = the push does not go, no exceptions without the surge commander's logged override. - GA hour zero: announcement goes out with the queue **already armed** — never arm it reactively after the spike lands. - Creator/partner embargo lifts are treated as pushes and scheduled in the same slots. ## Abort Criteria The surge commander executes; the release captain and executive on-call are informed. Each abort action is logged (`operator.launch.surge_abort.executed`). | Condition (sustained 15 min unless noted) | Action | | ----------------------------------------------------------- | ----------------------------------------------------------------------------- | | Session-start success < 98% globally | Freeze all marketing; halve global admission bucket | | Cold-join p95 > 8 s (budget is 5 s) | Halve global admission bucket; investigate before restoring | | BFF 5xx rate > 2% | Freeze admissions entirely; bypass lanes only; page BFF on-call | | ≥ 4 of 7 POPs in ≥ 85% backpressure simultaneously | Freeze admissions; open secondary-cloud capacity per the POP capacity drills | | Queue depth > 50,000 or median wait > 60 min | Switch overflow to waitlist mode: reuse the regional-waitlist landing (`REGION_ROLLOUT_COMPLIANCE.md`) as a global "we'll e-mail you a join link" capture; stop quoting wait times we cannot honor | | Redis queue state loss | Fail closed: all non-bypass traffic to waitlist capture, never open admission | | Any SEV-1 security incident (see security runbooks) | Marketing freeze automatic; admissions at surge commander's discretion | Recovery: each abort action unwinds one step at a time after its trigger metric is green for 30 minutes, in reverse order of application. Marketing resumes last, never first. ## Fail-Closed Criteria This plan is GA-blocking when any of the following holds: - the waiting room has not been load-tested at 3,300 auth requests/min (the 30x arrival assumption) with the queue armed, in staging, with evidence recorded; - the queue-to-backpressure wiring has no drill evidence showing zero admit-into-backpressure events during a simulated 90% POP condition; - the bypass lane for resume tokens or paid pre-warm attendees fails its drill; - the surge dashboard is not live or the marketing-gate signal is not wired to it; - the abort ladder has no named surge commander on the GA rota; - queue Redis failure does not fail closed to waitlist capture in the drill.