Telemetry, Performance, Testing & Release Gates

This page is V2's answer to a single question: how do you know the build is good enough to ship? For a cross-platform competitive game — the fighting roster on one side, the racing rulesets in V2/balance/racing on the other, both feeding a ranked ladder — that question has no single answer, only a composition of four interlocking systems. Telemetry is what the live game tells you; performance budgets are the per-platform frame-time contract that keeps 60 fps a competitive-integrity guarantee rather than a setting; testing is the automation that proves the deterministic simulation still behaves frame-for-frame; and the release gate composes all three (plus cert, canary, and crash-rate) into a go/no-go. The determinism that powers rollback is what makes this tractable: because a match is a pure function of its inputs, you can re-run it from a recorded input stream and assert the result hash exactly — a testing affordance most games simply do not have. This page is the quality spine, and it is deliberate about which parts are running C++ and CI, which are machine-checked policy, and which are documented obligations no build yet satisfies. The section hub is ../V2_ARCHITECTURE.md.

What ships, honestly#

The in-engine telemetry is real and substantial, not a contract scaffold: V2Telemetry (V2/ue/Source/V2Telemetry/, nine files, ~2,900 lines) implements the event taxonomy, the batcher, HMAC event signing, integrity heartbeats, and the PII/privacy/sampling policies as deterministic Blueprint libraries. The testing surface is real and large: V2Tests holds 407 *.spec.cpp suites across 442 files, including a literal 200-replay golden corpus re-simulated frame-by-frame on every run. The frame-budget and launch gates are real CI tooling — check-frame-budget.py and check-v2-launch-readiness.py both run in .github/workflows/v2-build.yml, not as prose.

Two honest qualifications carry through the page. First, the architecture claims "all Node packages under apps/v2/ declare @oshun/metrics and @oshun/tracing from day one"; the ground truth is 50 of 91 service package.json files declare them directly — the in-engine FV2ServiceObservabilityBinding is the contract that names the surface, but the per-package adoption is partial, and that gap is named here rather than papered over. Second, the launch-readiness gate is honestly fail-closed: its validator forbids the §138 checklist from marking anything [x] because no build, beta, cert pass, or drill exists, and the machinery is written so that red cannot be forged.

Telemetry & analytics#

The in-engine event spine#

V2's client telemetry is a registered, schema-validated taxonomy. UV2TelemetryBlueprintLibrary::BuildDefaultEventSchemas (V2/ue/Source/V2Telemetry/Private/V2TelemetryBlueprintLibrary.cpp:130) defines 39 event schemas, each an FV2TelemetryEventSchema carrying a category, a reliability class, its required fields, a bStripPII flag, and — critically — bAllowedDuringRollback = false (V2TelemetryTypes.h:95). The taxonomy spans the standard lifecycle (v2.player.session.started/ended, v2.match.level.loaded, v2.player.achievement.unlocked, v2.cosmetic.purchased, v2.player.ui.interacted, v2.player.error.reported) and the fighting-game-specific events that make balance legible: v2.match.combat.move.used, …damage.applied, …combo.ended, …resource.used, …finisher.triggered, …attack.blocked, and the release-gated v2.match.combat.frame-data.drift. Reliability is a real three-valued enum — BestEffort, Reliable, ReleaseGate — so a crash, a frame-budget breach, a cheat-detection signal, or a frame-data drift is tagged as a gate-quality event at the schema, not by convention.

The defining architectural choice is the destination. FV2TelemetrySenderConfig (V2TelemetryTypes.h:297) pins Endpoint = "event-bus://@oshun/event-bus" and EventBusPackageName = "@oshun/event-bus": V2 publishes to canonical v2.match.*, v2.cosmetic.*, and v2.player.* topics on the shared bus, not to a V2-private ingest. BuildEventBusPublishRecord (:689) stamps each event with its topic, source domain, schema version, and bMayInfluenceRollback = false. That single decision is what lets the analytics warehouse, the Maat balance team, and the Iris AI-commentary system all consume one stream without V2 reaching into anyone's database.

The batcher is a real algorithm, not a buffer that flushes on a timer. FV2TelemetryBatch::CanAddEvent (V2TelemetryTypes.h:347) admits an event only while the batch stays under both MaxEventsPerBatch (64) and MaxBatchBytes (64 KB), and UV2TelemetrySubsystem (V2TelemetrySubsystem.h:48) — a real UGameInstanceSubsystem — drives EnqueueEvent → Flush(reason) → CaptureRuntimeSnapshot, where the flush reason is the typed EV2TelemetryFlushReason (Interval / Size / Shutdown / Manual). A server-side variant, FV2ServerLogAggregationPolicy, rejects raw log lines (bRejectRawLogLines), requires structured game events, and mirrors them into the same event-bus batch — so dedicated-server logs and client telemetry land in one schema-validated pipe.

Signed events and integrity heartbeats#

Telemetry that feeds anti-cheat must be tamper-evident, and V2 implements that with real cryptography rather than a flag. SignTelemetryEvent (:494) builds a canonical, attribute-sorted payload and signs it with ComputeHmacSha1Hex (:90), which calls FSHA1::HMACBuffer — genuine HMAC-SHA1 over the UTF-8 payload, not a hash placeholder. VerifySignedTelemetryEvent (:512) recomputes the MAC, checks the session/key id, and — when a v2.match.replay.hash event fails verification — sets bReplayEventTampered, the signal the integrity layer escalates on. BuildIntegrityHeartbeat / VerifyIntegrityHeartbeat (:554 / :578) extend this to a per-session heartbeat that binds the build hash, content version, and replay-seed checksum; a mismatch sets bIntegrityHeartbeatDesync, catching a client whose simulation has diverged from the authoritative build. Every one of these structs carries bMayInfluenceRollback = false — the integrity-telemetry path is observable but can never perturb the deterministic sim it observes.

Ingest, PII, sampling, and the balance dashboards#

FV2TelemetryIngestPipelineConfig declares the warehouse path as an explicit stage list — Firehose → PiiStrip → Warehouse — with bPiiStripAtIngest and bRejectRawAccountIdentifiers set, so raw account ids never reach analytics; the full PII-allowlist contract lives in Security & Compliance. FV2TelemetryPrivacyPolicy makes bOptOutPersistsAcrossSaveReset a structural property (a save wipe cannot silently re-enable collection), and FV2GhostDataSharingPolicy keeps ghost capture bOptInRequired. Sampling is policy-bounded but safe: FV2TelemetrySamplingPolicy randomizes a per-session sample percentage yet forces bCrashEventsAlwaysSampled and bCheatDetectionEventsAlwaysSampled, so the gate-quality events are never sampled away.

The payoff is the balance dashboard set: BuildDefaultBalanceMetrics (:193) defines 12 FV2BalanceDashboardMetrics — move_pick_rate, win_rate_on_hit, win_rate_on_block, frame_data_drift, the hot_cold_fighter_heatmap, finisher_trigger_rate, rage_quit_rate, frame_budget_p99, and the player funnels (match_acceptance_funnel, per_fighter_retention, churn_risk_weekly) — each wired to a Grafana URI and a source event. Because frame-data.drift events compare authoring frames to runtime frames, the same dataset the data pipeline exports as frame-data.csv is what the dashboard argues from, and an A/B framework (FV2BalanceExperimentPolicy, bRollbackRequired = true) lets a balance change be canaried with a rollback runbook before it goes wide.

Performance budgets#

The per-platform frame-time contract#

The frame-time contract is per-platform and explicit, because frame pacing is input timing in a competitive game:

The per-frame split is a hard budget, and it exists as data, not prose: BuildDefaultQAReleaseGateFeatureSet (V2/ue/Source/V2Tests/Private/V2AutomationHarness.cpp:404) populates SystemFrameBudgets as gameplay 4 ms, animation 3 ms, render 8 ms, audio 1 ms, network 0.5 ms, and incremental GC 1 ms; LoadingBudgets bound cold-boot to 30 s on PS5 and a mid-session match load to 8 s; and a LatencyBudget block pins input-to-render at 35 ms @ 60 fps / 25 ms @ 120 fps with a 16.6 ms sim tick and a 2 ms controller poll, flagged bMeasuredWithControllerCameraRig so the number is an end-to-end measurement, not a synthetic estimate. FV2CrashPerfPolicy mirrors the 16.667 ms frame budget against the per-platform budget ids (PC.60Hz, PS5.60Hz, XSX.60Hz, Switch2.60Hz) and emits v2.match.performance.frame-budget-exceeded when a frame blows it.

The frame-budget gate is real CI#

The budget is enforced, not merely documented. V2/ue/Tools/check-frame-budget.py (324 lines) parses Unreal JSON/CSV profiling exports, identifies combat frames by marker (phase: Combat, combatFrame: true, a scope/stat containing V2Combat) or by being nested under a combat-labeled collection, reads the worst millisecond field from each sample, and fails the build on any combat frame over 16.6 ms. It ships a --self-test mode that proves a good fixture passes and an over-budget fixture fails — so the gate's own correctness is regression-tested. Both run in CI: .github/workflows/v2-build.yml invokes check-frame-budget.py --self-test (line 2312) and the real profiling run (line 2360) after automation, on the Win64 reference runner, per V2/docs/ci/frame-budget-gate.md. A regression past budget fails the PR rather than shipping a stutter.

Testing, QA & cert#

The automation surface#

The UE automation layer is V2Tests — 407 *.spec.cpp suites (30 in Automation/, the rest under Private/) across 442 files — covering GAS, the motion parser, the hitbox engine, rollback determinism, save round-trip, and data-table integrity: precisely the subsystems where a silent regression is a balance or fairness bug rather than a cosmetic one. Services.Module.spec.cpp asserts the service surface directly (:45): 62 required Oshun domain adapters and exactly 2 observability bindings, each verified to name @oshun/metrics and @oshun/tracing, to be bRequiredFromDayOne, and to stay off rollback.

Golden-replay regression — the determinism dividend#

GoldenReplayCorpus.spec.cpp (V2/ue/Source/V2Tests/Private/Netcode/) is the load-bearing test, and the one a non-deterministic game cannot write. The corpus is 200 deterministic input-stream replays (constexpr int32 CorpusSize = 200) persisted as JSON manifests under Content/V2/Tests/GoldenReplays/Corpus/, eight gameplay archetypes (neutral walk, eight-way run, stance cycling, heat/ki charge, power-crush guard, pressure strings, hit exchange, juggle carry), each storing its exact per-frame input stream plus state hashes sampled every 10 frames and a final hash. The regression re-runs every manifest through FV2SimWorld and fails on the first divergent frame — so a rendering or logic change that perturbs the sim by even one frame is caught before it ships. The generator runs only under the explicit -V2GenerateGoldenCorpus flag, self-verifies each manifest by an immediate re-simulation before writing, and never overwrites silently; cross-process stability of FV2SimWorld::ComputeStateHash is itself part of what the regression asserts. The release-gate feature set fixes the contract at RequiredReplayCount = 200, FramesPerReplaySample = 600, bFrameByFrameDivergenceDetection = true, bRunsPerBuild = true, and bNightlyAllCookedBuilds = true.

Gauntlet, cert & dogfood#

Above the unit/replay layers, BuildDefaultGauntletDrivers (V2AutomationHarness.cpp:223) assembles full-game smoke suites (launch, tutorial, arcade ladder, online-headless 1v1) that run on every PR. The cert surface is enumerated as data in the same feature set: CertCompliance.PlatformCertIds lists PlayStation.TRC, Xbox.XR, Nintendo.Lotcheck, SteamDeck.Verified, and MicrosoftStore.Cert, with bAntiCheatCompatibilityCertPerPlatform and a trophy/achievement plan that AvoidsForcedGrind. BetaDogfood declares a 30-day internal closed beta, a pre-launch network-test weekend, and a press-playable feature freeze. These are specifications the launch gate below composes — real structured policy, but policy whose teeth are the validators, not a runtime that has executed them.

Observability & the release gate#

The shared spine, and an honest coverage gap#

V2's intended posture is that every backend surface emits through the shared platform libraries. The in-engine contract is real: FV2ServiceObservabilityBinding (V2/ue/Source/V2Services/Public/V2ServiceTypes.h:83) and BuildRequiredObservabilityBindings (V2OshunDomainAdapters.cpp:121) produce two bindings — v2-services-node over apps/v2/* and v2-services-unreal over V2/ue/Source/V2Services — each pinning @oshun/metrics / @oshun/tracing, bRequiredFromDayOne = true, and bMayInfluenceRollback = false, and the Services automation spec enforces all of it. The shared libraries are themselves substantial: @oshun/metrics (libs/shared/metrics/src/registry.ts, 416 lines) is a real prom-client registry, and @oshun/tracing ships a tracer, propagation, decorators, and middleware.

The honest qualification is coverage. The architecture asserts all Node packages under apps/v2/ declare the spine "from day one"; in fact 50 of 91 package.json files declare @oshun/metrics and @oshun/tracing directly — the data-pipeline and analysis libs (e.g. @v2/ab-test-analysis-pipeline) compose domain packages instead. So the accurate framing is: the binding is the day-one contract and the in-engine test enforces it, while per-package adoption across the 91 services is partial — a real gap, named rather than implied away. Beyond the spine, observability includes a symbolicated crash reporter, in-the-loop dev fault injection (bDevFaultInjectionEnabled), per-platform perf dashboards, the network-quality dashboard (matchmaking p99, rollback-frame distribution, packet loss, rage-quit rate), an anti-cheat false-positive dashboard, and canary-analysis automation for the online services.

The launch-readiness gate (honestly fail-closed)#

The release gate is the composition — regression suite + golden replays + frame-budget + crash rate + cert pass + RTC pass + canary pass — and §138 Launch Readiness aggregates it into one go/no-go. Its data lives at V2/balance/launch/launch-readiness.json (schema v2.launch.readiness-data.v1): 60-day internal dogfood across PC/PS5/XSX/Switch2, a 5× network-stress multiplier, a five-dashboard monitoring set, a canary/rollback plan, 27 critical journeys, 33 feature gates, and 21 exit criteria (60 fighters, 30 stages, 100 ms playable RTT, a 30 ms local-feel reference, and a noComingSoonPanelsVisible proof).

What makes this honest is the validator. V2/ue/Tools/check-v2-launch-readiness.py (404 lines, run in CI at v2-build.yml:2125) cross-checks the contract, the data, the C++ catalog source, and the docs — and then fails closed on the TODO. Its embedded rule (lines 327–339) is blunt: because no build, deployment, beta, cert, or drill exists, the §138 checklist rows must stay [ ]/[~], and any [x] is treated as a fabricated completion unless it is on an explicit evidence allowlist with committed proof on disk. Today that allowlist holds exactly one row — the §6.7 per-move verification, backed by the on-box Linux-editor automation report (V2/docs/qa/automation-runs/2026-06-12-linux-editor-report.json, requiring a Success run of V2.Combat.MoveData.AssetContract). Every other launch row is held red by construction. The gate is therefore not green and the tooling refuses to let it be hand-edited green — the correct posture for a launch gate over a game that has not yet been built, beta'd, or certified.

Edge cases and failure modes#

• Telemetry can never perturb the sim. Every schema sets bAllowedDuringRollback = false and every publish/binding/policy struct sets bMayInfluenceRollback = false; observability rides strictly off the deterministic path it observes.
• Tampered replay-hash telemetry is flagged, not trusted. A v2.match.replay.hash event with a bad HMAC trips bReplayEventTampered, and a heartbeat whose build hash / content version / replay-seed checksum diverges trips bIntegrityHeartbeatDesync — real HMAC-SHA1 verification, escalated to the integrity layer in Online Backbone & Competitive Integrity.
• Opt-out survives a save wipe; crash/cheat events survive sampling. bOptOutPersistsAcrossSaveReset and the always-sample overrides are structural, so privacy and gate-quality signals are not silently lost.
• One divergent frame fails the build. The golden corpus re-simulates through FV2SimWorld and stops on the first hash mismatch — the determinism dividend the combat and rollback layers earn.
• The launch gate fails closed, and its red is the point. The validator rejects any §138 [x] without committed automation evidence, so a build cannot be declared launch-ready by editing a checklist.

Where this connects#

• Up to what it measures: Combat: GAS, Frame Data & Determinism and Rollback Netcode & Tag-Team — the determinism the golden-replay regression and frame-data drift metric exploit.
• Sideways: Build, Cook, Assets, Data & Production (the cook frame-budget gate and the frame-data.csv export the dashboards read) and Security, Compliance & Sister-Monorepo Integration (the PII allowlist applied at telemetry ingest).
• Integrity: Online Backbone & Competitive Integrity — where signed telemetry and integrity heartbeats feed anti-cheat and the replay-hash validator.
• Platform: the shared platform observability libraries (@oshun/metrics, @oshun/tracing) every V2 service composes.