Fighting Game · Guides & deep dives

V2 Dedicated Server Operations Runbook

Server artifacts are produced by the BuildGraph server binary compilation pipeline:

4sections4 minread
Markdown source
On this page

This runbook is the operational contract for deploying, monitoring, and troubleshooting V2Server dedicated server builds. It covers TODOS.phase-72.72.2.1.15 and ties together the server build, health, budget, logging, replay, hot-patch, integration, and determinism gates from phase 72.

Deployment#

Artifacts#

Server artifacts are produced by the BuildGraph server binary compilation pipeline:

  • LinuxServer-x86_64
  • LinuxServer-ARM64
  • Win64Server

Each artifact is built from target V2Server in Shipping configuration with server-only cooking, pak archives, deterministic floating-point settings, and no client payload. Promotion requires the binary contract V2/ue/Content/V2/Netcode/Server/ServerBinaryCompilationPipeline_V2_Contract.json.

Preflight#

Run these gates before any artifact can receive player traffic:

bash
python3 V2/ue/Tools/check-v2-server-binary-compilation-pipeline.py
python3 V2/ue/Tools/check-v2-server-tick-rate-config.py
python3 V2/ue/Tools/check-v2-server-health-monitoring.py
python3 V2/ue/Tools/check-v2-server-budget-enforcement.py
python3 V2/ue/Tools/check-v2-server-log-aggregation.py
python3 V2/ue/Tools/check-v2-server-replay-recording.py
python3 V2/ue/Tools/check-v2-server-hot-patching.py
python3 V2/ue/Tools/check-v2-server-integration-test-harness.py
python3 V2/ue/Tools/check-v2-server-determinism-verification.py

The release owner records the artifact id, Git SHA, platform id, config revision, region, and rollback artifact id in the deployment ticket before opening matchmaking.

Boot Contract#

Dedicated server launch must include a region id, match id, supported tick rate, and headless rendering:

bash
V2Server /Game/Maps/L_OnlineHeadless -server -log -unattended -nullrhi \
  -NoSound -v2region=us-east -v2matchid=<match-id> -v2tickrate=60

Supported tick rates are 30, 60, and 128. Default production boot is 60 Hz; 128 Hz requires an explicit competitive-mode allocation, and 30 Hz is the recovery profile for overloaded or low-population sessions.

Rollout#

Roll out new server binaries by canarying new allocations first:

  • Start at 5 percent of new sessions for one region.
  • Block expansion while bServerHealthEligibleForNewSessions is false for any canary host.
  • Drain old hosts with new-session admission disabled, then use graceful player migration for active sessions.
  • Promote only after health, budget, replay, log aggregation, integration harness, and determinism checks are green for the canary artifact.
  • Roll back by disabling new allocations for the candidate artifact, draining live sessions, and restoring the previous V2Server artifact id.

Monitoring#

Health Dashboard#

The live dashboard must display the health state from FV2ServerHealthReport, including these structured signals:

  • ServerHealth.TickOverrun
  • ServerHealth.MemoryHigh
  • ServerHealth.PlayerCap
  • ServerHealth.CrashDetected
  • ServerHealth.CrashHeartbeatStale

Default thresholds are 20 ms tick overrun, 3 consecutive tick overruns, 8192 MiB memory usage, 128 players, and 15 seconds maximum crash heartbeat age. Any degraded or crash-suspected host is ineligible for new sessions.

Budget Dashboard#

Budget enforcement publishes CPU, memory, garbage collection, drain, and kill signals:

  • ServerBudget.CpuOverrun
  • ServerBudget.MemorySoftLimit
  • ServerBudget.MemoryHardLimit
  • ServerBudget.AggressiveGarbageCollection
  • ServerBudget.NewSessionDrain
  • ServerBudget.RunawayServerKill
  • CPU utilization target: at or below 90 percent
  • game thread target: at or below 16.6 ms
  • server tick target: at or below 20 ms
  • aggressive garbage collection: starts at 7168 MiB
  • soft memory limit: 8192 MiB
  • hard memory kill threshold: 12288 MiB
  • process exit threshold: 10 consecutive CPU overruns

When the policy requests DrainNewSessions, matchmaking must stop placing new players on the host. When the policy requests TerminateRunawayServer, crash diagnostics and the final structured event batch must be preserved first.

Logs, Replays, And Profiles#

Structured server events publish only through the event bus topics v2.match.server.lifecycle, v2.match.server.player-session, v2.match.server.authoritative-state, and v2.match.server.admin-command. Raw log lines stay out of aggregation.

Replay recording must capture all authoritative inputs needed for dispute resolution, regression, and determinism reproduction. Performance profiling mode is enabled only for investigation windows and must keep rollback state isolated from profiling and telemetry side effects.

Alerts#

Page server on-call when any of these conditions persist for two consecutive samples:

  • crash heartbeat stale or crash detected
  • memory above 8192 MiB
  • CPU above 90 percent
  • tick overrun count at or above 3
  • player count at or above 128
  • log aggregation shutdown flush failure
  • determinism final state hash mismatch

Troubleshooting#

Host Fails Preflight#

Rerun the preflight gate that failed, inspect the associated contract JSON, and compare the artifact id to the deployment ticket. Common causes are a stale server binary, unsupported tick rate, unsigned hot-patch manifest, or a platform artifact that was built for the wrong architecture.

New Sessions Stop Landing#

Check bServerHealthEligibleForNewSessions, bServerBudgetNewSessionsDrained, and ServerHealth.PlayerCap. If the host is healthy but drained, clear the drain only after CPU, memory, player count, and crash heartbeat signals are green.

Tick Or CPU Overruns#

Confirm the current tick rate and inspect ServerHealth.TickOverrun, ServerBudget.CpuOverrun, and the performance profiling capture. For overloads, move new allocations away from the host, use the 30 Hz recovery profile for eligible sessions, and preserve the profiler artifact before terminating a runaway process.

Memory Pressure#

If memory crosses 7168 MiB, confirm aggressive garbage collection was requested. If it crosses 8192 MiB, drain new sessions. If it reaches 12288 MiB, preserve crash diagnostics, publish the final structured batch, and let the budget policy terminate the server.

Crash Heartbeat Failure#

Crash heartbeat age above 15 seconds moves the host to crash-suspected state. Block new sessions, collect crash diagnostics, keep replay artifacts attached to the match id, and migrate active players through the graceful shutdown flow when the process still responds.

Hot Patch Rejected#

Verify the manifest signature, monotonic revision, payload size below 65536 bytes, and config domain. Runtime hot patches are limited to ServerConfig.BalanceTweaks and ServerConfig.FeatureFlags; changes requiring a restart must ship through a new binary rollout.

State Divergence#

Collect the replay id, input stream hash, final state hash, server artifact id, map path, tick rate, and match seed. Reproduce with the server integration test harness first, then run server determinism verification with the same input stream. A final state hash mismatch blocks promotion until the rollback or authoritative-state change is fixed.

Log Aggregation Backlog#

Confirm event bus connectivity and the aggregation batch limits: 128 events, 262144 bytes, and 2 seconds flush interval. During shutdown, the final flush must complete before the host exits or migrates players.

Verification#

The static contract lives at V2/ue/Content/V2/Netcode/Server/ServerOperationsRunbook_V2_Contract.json. Repository validation is enforced by V2/ue/Tools/check-v2-server-operations-runbook.py, plus the phase 72 server checkers listed in the preflight section.