plan-id: v5-capacity-load.v1
Status: Draft for review — fills audit gap V1_V7_PLAN_SET_AUDIT_2026-06-12 §6.2 ("no concurrency/CCU targets or load-test plan for a 64-player persistent world; GameLift+EKS named, never sized").
Owners: V5 Capacity Lead (SRE) — fleet sizing, autoscaling, cost model. V5 Online Services Lead — per-service SLOs. V5 Netcode Lead — game-server session budgets and load-test pass criteria. Live-Ops Producer — CCU forecast sign-off.
Grounding: V5/V5_DEPENDENCIES.md §16 (AWS multi-region us-east / us-west /
eu-west / ap-northeast / ap-southeast + Alibaba CN; GameLift game servers; EKS;
MSK Kafka; ClickHouse on EKS), V5/V5_ARCHITECTURE.md §"Online Services
Backend" (16 services), V5/V5_features.md §"Online Services & Multiplayer
Infrastructure" (64-player Bureau HQ dedicated hub, 80 ms matchmaking ping
ceiling, 5%/20% packet-loss envelope, sub-2 s host migration, 500-event/2 KB
client telemetry buffering), launch date 2027-03-01 per
V5/live-service/season-1-live-service-manifest.json.
1. CCU Targets#
1.1 Launch-peak derivation from open-world comparables#
Planning assumption adopted 2026-06-12 — comparable Steam launch-peak concurrents (public SteamDB figures, rounded):
| Title (class) | Steam launch peak CCU |
|---|---|
| Cyberpunk 2077 (established-studio IP) | ~1,050,000 |
| Elden Ring (established IP) | ~953,000 |
| Hogwarts Legacy (licensed mega-IP) | ~880,000 |
| Starfield (established studio, new IP) | ~330,000 |
| Dying Light 2 (sequel, mid-tier) | ~275,000 |
| New-IP AAA open-world median | 100,000–300,000 |
V5 is a new IP. The mega-IP comparables are ceilings, not bases. Base case: Steam peak of 60,000 CCU (lower half of the new-IP band, reflecting no franchise pull, offset upward by the five-cells-in-one-box pitch).
V5 ships on 9 platforms with full crossplay. Planning assumption adopted 2026-06-12: Steam ≈ 40% of total concurrents for a crossplay multiplatform title (PC-heavy genre mix pulls Steam share up; console install base pulls it down).
Global launch peak CCU = 60,000 / 0.40 = 150,000 CCU
Design point (1.5x surge headroom) = 225,000 CCU
Sustained launch-month average (peak:avg 2.5:1) = 60,000 CCU
Post-launch steady state (month 4+, 30% of peak) = 45,000 CCU
Targets: provision for 225k CCU, alert-test at 150k, cost-model at 60k average.
1.2 Online concurrency mix#
V5 is offline-capable and single-player heavy (features§"V5 Product Promise");
not all CCU touch multiplayer. Planning assumption adopted 2026-06-12, to be
recalibrated against closed-beta telemetry (see
V5/docs/planning/beta-program.md):
| Slice of CCU | Share | At 150k peak |
|---|---|---|
| Offline / online-but-solo (services-light: auth, telemetry, sync) | 65% | 97,500 |
| Bureau HQ 64-player hub presence | 14% | 21,000 |
| Dedicated-server MP (Urban Freeroam, Frontier Posse, Capital Raid, sci-fi PvP, twitch PvP) | 16% | 24,000 |
| Listen-server co-op + LAN (no game-server cost) | 5% | 7,500 |
Online-session CCU = 35% of total = 52,500 at peak / 78,750 at design point.
1.3 Regional split#
Planning assumption adopted 2026-06-12 (comparable AAA crossplay titles): us-east 30%, eu-west 30%, us-west 15%, ap-northeast 15%, ap-southeast 10%. CN (Alibaba Cloud) is sized separately by the CN publishing partner and is out of scope here.
2. GameLift Fleet Sizing — 64-Player Dedicated Hubs#
2.1 Per-session budget#
Planning assumption adopted 2026-06-12, validated as a hard gate in technical alpha (beta-program.md §Technical Alpha): one UE5.5 dedicated hub server session (64 players, 30 Hz server tick, Iris replication per ADR 0014) budgets:
- 4 vCPU sustained (server frame p99 ≤ 33.3 ms at 64 players)
- 6 GB RSS (hub world + 64 player states + workshop data-only mod sets)
- Egress ≈ 16.4 Mbps/session (64 players × 256 kbps replication budget)
2.2 Sessions per instance#
Instance type: c6i.12xlarge (48 vCPU, 96 GB) — GameLift-supported, best vCPU-$/perf in the c6i family at this size.
CPU bound: floor((48 - 2 reserved for OS/agent) / 4) = 11 sessions
Memory bound: floor((96 - 8 reserved) / 6) = 14 sessions
Network: 10 x 16.4 Mbps = 164 Mbps << instance cap — not binding
Packing cap (scheduling + noisy-neighbor headroom): 10 sessions/instance
10 hub sessions (640 player-slots) per c6i.12xlarge.
Small-mode servers (16-player rooms: Capital Raid squads, twitch PvP lobbies, private rooms from the server browser): 1 vCPU / 1.5 GB per session → packing cap 40 sessions (640 slots) per c6i.12xlarge (same slot density; memory bound 58, CPU bound 46, capped at 40 for headroom).
2.3 Fleet math at design point (225k CCU)#
Hub players = 225,000 x 14% = 31,500 → 31,500/64 = 493 sessions
Hub instances = ceil(493 / 10) = 50 → +20% buffer = 60 instances global
Dedicated-MP = 225,000 x 16% = 36,000 → 36,000/16 = 2,250 sessions
MP instances = ceil(2,250 / 40) = 57 → +20% buffer = 69 instances global
Total game-server fleet at design point ≈ 129 c6i.12xlarge, split by §1.3:
us-east 39, eu-west 39, us-west 19, ap-northeast 19, ap-southeast 13
Buffer policy: 20% available-session buffer at all times (GameLift
target-tracking on PercentAvailableGameSessions = 20). FleetIQ Spot/On-Demand
mix 60/40 — Spot interruption is acceptable for small-mode rooms (players
re-matchmake), On-Demand pinned for hub sessions (64-player presence must not
evaporate; hub sessions are placed only on the On-Demand portion).
First-party relays (PSN, XBL, Steam Datagram Relay) front platform-pool sessions per features§"Online Services" degradation rules; the GameLift fleet above carries all cross-platform sessions and is sized as if it carried 100% — relay offload is upside, not a dependency.
3. Per-Service Capacity SLOs (16 services)#
All rates at the 225k design point. Availability SLOs are monthly. Latency = p99 server-side. Derivations noted inline; every service is horizontally scaled on EKS (HPA, see §4).
| Service | Peak load (derivation) | p99 | Availability | Degradation behavior (per spec) |
|---|---|---|---|---|
| auth | 250 RPS login (patch-storm: 50% of CCU reconnect in 10 min = 112,500/600 = 188/s, rounded up) + 250 RPS refresh (15-min JWT: 225k/900 s) | 300 ms | 99.95% | Outage drops players to offline-deterministic play |
| friends | 3,750 RPS presence (1 update/min/online user = 225k/60) | 400 ms | 99.9% | Stale presence acceptable; no play impact |
| parties | 400 RPS (≈5% of online users in party ops per minute) | 400 ms | 99.9% | Party features unavailable; solo queue still works |
| matchmaking | 150 tickets/s (re-queue churn: 2,250 sessions / 20-min avg = 1.9 sessions/s x 16 players = 30/s steady; 5x surge factor) ; TTM p95 < 60 s | 500 ms | 99.9% | Clear retry state, never a hang (features§"Online Services") |
| leaderboards | 30 writes/s (match completions incl. anti-cheat re-validation) + 500 reads/s | 250 ms | 99.5% | Score posts queue; reads serve cached |
| replays | 5 uploads/s x 20 MB (≈10% of completed matches) → S3; 50 RPS metadata | 1 s | 99.5% | Upload retries client-side |
| telemetry | 112,500 events/s (225k x 0.5 events/s/client); 7,500 batch-POSTs/s (30 s batching) | 800 ms | 99.9% ingest | Client buffers 500 events / 2 KB per batch, flushes on recovery |
| anti-cheat | 2,600 signals/s (78,750 online x 2 signals/min); ML classifier consumes Kafka stream | async | 99.5% | Fail-open to post-match re-validation; never blocks play |
| crash-reporting | 10 dumps/s sustained, 100/s after a bad patch (sizing case) | 2 s | 99.0% | Client retries with backoff |
| companion-app-bridge | 2,250 RPS (≈5% of CCU concurrently in app x 0.2 RPS) | 400 ms | 99.5% | Read-only; app shows stale data |
| mindpalace-cloud-sync | 500 RPS (sync on device/session transition) | 600 ms | 99.9% | Blocks cross-device pickup, not play |
| workshop | 1,500 RPS catalog browse (downloads offloaded to CloudFront); 5 publishes/s | 500 ms | 99.5% | Subscribed content already local; browse-only outage |
| compliance-dsar | Batch; 100 requests/day | n/a | 99.0% | SLA measured in days (30-day GDPR clock), not ms |
| live-service-calendar | 2,000 RPS (edge-cached ≥95% hit ratio; origin ≤100 RPS) | 200 ms | 99.9% | Clients cache last-known calendar |
| balance-ledger | 300 RPS public reads (cached); <1 write/s | 300 ms | 99.5% | Read-only public surface |
| faction-rep | 200 RPS (Sci-Fi cell writes on mission resolution) | 400 ms | 99.5% | Buffered in cell save; reconciles on recovery |
Error-budget policy: a service that burns >50% of its monthly budget in 7 days freezes feature deploys for that service until burn returns under curve (Live-Ops SRE Lead owns enforcement).
4. Autoscaling Policies#
GameLift fleets (per region):
- Target-tracking:
PercentAvailableGameSessions = 20%. - Scale-out: available sessions < 15% for 2 consecutive minutes → add instances (step: +10% of current fleet, min 2 instances). Instance warm-up budget 8 minutes (AMI bake includes server binary; no download at boot).
- Scale-in: available sessions > 30% for 20 consecutive minutes → remove instances with session protection on (instance drains; never kills an occupied hub).
- Floor: launch week floor = 50% of design-point fleet held warm regardless of load (absorbs the 00:00 UTC 2027-03-01 spike faster than scale-out can).
- Spot interruption: small-mode rooms only; hub sessions On-Demand-pinned (§2.3).
EKS services: HPA on CPU 60% target plus per-service custom metric (RPS per pod at 70% of load-tested ceiling). Min replicas = 3 per region (zonal spread). Cluster Autoscaler headroom: 15% spare node capacity. Telemetry edge service additionally scales on Kafka producer-buffer depth (>10 s of lag → scale out).
ClickHouse / Kafka: not autoscaled. Provisioned to design point (§5), capacity-reviewed monthly by the V5 Capacity Lead; expansion is a planned change, not an autoscaling event.
5. Load-Test Plan#
Tooling: k6 (HTTP service load, distributed runners on a dedicated EKS
load cluster); custom UE Gauntlet headless bot clients ("GauntletLoad") for
game-server fill — bots run real client netcode and replication, scripted
movement/combat loops per cell; tc netem on bot hosts for latency/loss
injection; GameLift FlexMatch latency simulation for matchmaking-policy tests.
All runs against a production-shaped staging environment in us-east + eu-west
(2-region minimum to exercise cross-region paths).
Scenarios — each runs green twice before the open-beta stress weekend and once on the launch-candidate build:
| ID | Scenario | Pass criteria (tied to specced envelopes) |
|---|---|---|
| LT-1 | Login storm — 250 RPS login + 250 RPS refresh for 30 min (patch-day reconnect, §3) | auth p99 ≤ 300 ms; error rate < 0.1%; zero cascading failure into friends/parties |
| LT-2 | Hub fill — 500 hub sessions × 64 bots (32,000 bots), 2-hour soak | Server frame p99 ≤ 33.3 ms per session; zero session crashes; RSS ≤ 6 GB/session; egress ≤ 16.4 Mbps/session |
| LT-3 | Matchmaking surge — 150 tickets/s with simulated regional ping spread up to 80 ms | TTM p95 < 60 s; no match formed above the 80 ms ping ceiling (features§"Online Services"); retry state (never hang) when queue saturated |
| LT-4 | Host-migration storm — 1,000 concurrent listen-server co-op sessions; kill 10% of hosts within a 60 s window, repeat 5 waves | Migration success ≥ 99.5%; migration time p95 < 2 s (features§"Network Quality, LAN, and Server Browser"); no save corruption in any migrated session |
| LT-5 | 20%-loss degradation — netem 20% loss + 80 ms RTT on 25% of bots in twitch-PvP and open-world MP sessions | Twitch PvP stays within 8-frame rollback window without desync; open-world MP hit validation stays lag-compensated; session remains playable at 5% loss and degrades gracefully (no disconnect cascade) at 20% per arch§"Observability, Evaluation, Release Gates" |
| LT-6 | Telemetry flood — 112,500 events/s for 1 hour, then kill ingestion for 10 min, then recover | Zero event loss end-to-end (client 500-event/2 KB buffers flush on recovery); Kafka consumer lag drains < 15 min after recovery |
| LT-7 | Replay surge — 20 uploads/s × 20 MB for 15 min (4× nominal) | Upload success ≥ 99%; no impact on matchmaking/leaderboard SLOs (bulkhead check) |
| LT-8 | Full-region failover — drain us-east at 60% design load | Surviving regions absorb load within autoscaling SLAs; matchmaking re-homes within 5 min; error budget burn < 10% of monthly during the drill |
A failed scenario blocks launch the same way a Launch Readiness gate does (features§"Launch Readiness"); LT results attach to the launch-readiness evidence bundle.
6. Infrastructure Cost Model (skeleton)#
All prices are us-east-1 list-price snapshots, planning assumption adopted 2026-06-12; the model's purpose is structure and order-of-magnitude, refreshed quarterly by the V5 Capacity Lead with Finance.
Game servers (GameLift)
c6i.12xlarge on-demand ≈ $2.04/hr; Spot ≈ 30% of on-demand ≈ $0.61/hr
Blended (60% Spot / 40% OD) = 0.6 x 0.61 + 0.4 x 2.04 = $1.18/hr
Player-slots per instance = 640 (both hub and small-mode packing, §2.2)
Cost per occupied player-hour = 1.18 / (640 x 0.70 occupancy x 0.83 buffer factor)
= 1.18 / 372 ≈ $0.0032
Share of CCU on dedicated servers = 14% hub + 16% MP = 30%
Game-server cost per total CCU-hr = 0.30 x 0.0032 ≈ $0.00096 ≈ $0.0010
ClickHouse on EKS (telemetry warehouse)
Avg ingest at 60k CCU = 60,000 x 0.5 events/s x 350 B = 10.5 MB/s
Raw/day = 10.5 x 86,400 ≈ 907 GB; compressed (10:1) ≈ 91 GB/day
90-day hot retention ≈ 8.2 TB → provision 12 TB gp3
Compute: 6 x m6i.4xlarge x $0.768/hr = $4.61/hr ≈ $3,365/mo
Storage: 12,288 GB x $0.08/GB-mo ≈ $983/mo
Total ≈ $4,350/mo → / (60,000 CCU x 730 hr) ≈ $0.0001 per CCU-hr
Kafka (MSK)
Peak ingress 39.4 MB/s (design point) x 3x replication ≈ 118 MB/s cluster write
6 x kafka.m5.2xlarge x $0.84/hr = $5.04/hr ≈ $3,680/mo + storage ≈ $4,500/mo
≈ $0.0001 per CCU-hr
Databases + cache (16 services)
Avg per service (RDS multi-AZ r6g mix + share of ElastiCache) ≈ $790/mo
16 x $790 ≈ $12,600/mo ≈ $0.0003 per CCU-hr
CDN + S3 (replays, workshop, patches — excl. launch-day patch spike)
≈200 TB/mo egress x $0.06/GB committed ≈ $12,000/mo + S3 ≈ $2,000/mo
≈ $0.0003 per CCU-hr
EKS service compute + observability stack ≈ $9,000/mo ≈ $0.0002 per CCU-hr
Multi-region adder (cross-region replication, NAT, inter-AZ, 5 regions operating floors): +15% on the subtotal.
Subtotal per CCU-hr = 0.0010 + 0.0001 + 0.0001 + 0.0003 + 0.0003 + 0.0002
= $0.0020
With multi-region = 0.0020 x 1.15 ≈ $0.0023 per CCU-hour
Launch-month (60k avg CCU): 60,000 x 730 x 0.0023 ≈ $101,000/mo
Steady state (45k avg CCU): ≈ $76,000/mo
Launch-week design-point burst adds ≈ $25k one-time (fleet floor at 50%, §4)
Out of scope for this skeleton (tracked by Finance with the Live-Ops Producer):
CN region (partner-borne), voice infrastructure (provider-dependent, see
V5_DEPENDENCIES.md), bandwidth for the day-one patch, and people costs.
7. Review cadence#
- Recalibrate §1 mix and §2 session budgets after technical alpha (2026-10), again after closed beta (2026-12) — owner: V5 Capacity Lead.
- LT-1..8 green twice before the open-beta stress weekend (2027-01-28) and once on the launch candidate — owner: V5 Netcode Lead.
- Cost model refresh quarterly and at each beta gate — owner: V5 Capacity Lead
- Finance.