V4 ships six genuinely different games — a Rainbow-Six breach, a Hitman sandbox, a Commandos squad-tactics puzzle, a Black-Myth boss duel, an Age-of-Empires ladder, and a couch Contra run — onto ten performance targets that range from PC Ultra at 4K down to an Android phone. Two questions decide whether that catalogue is shippable and operable: what is the live game telling us? and is every cell still inside its frame-time and memory contract on every platform? This page is about the three systems that answer them — telemetry (the event stream the client emits and the backend ingests), observability (how that stream and the service fleet are watched in production), and performance budgets (the per-cell, per-platform contract a CI gate holds the build to). The honest theme, the same one the rest of V4 follows, is that some of this is real C++/Rust on disk with passing tests, and some is specified policy that a manifest and a gate enforce rather than a runtime object — and this page draws that line explicitly so you know which is which. The hub for the V4 architecture set is ../V4_ARCHITECTURE.md.
What ships, honestly#
The client telemetry subsystem is real: V4/ue/Source/V4Telemetry is a
fully-implemented UGameInstanceSubsystem (UV4TelemetrySubsystem, ~770 lines
of .cpp) with schema validation, a byte-bounded batched sender, an offline
disk queue, region opt-out, and an HTTPS-plus-certificate-pin transport policy —
exercised by a real automation test,
V4.Telemetry.Runtime.QueuePrivacyTransport. The ingest path is real Rust:
apps/v4/telemetry-ingest writes batched INSERT … FORMAT JSONEachRow rows to
ClickHouse over HTTP and reports not_configured honestly when no endpoint is
set — no fabricated success. The performance budgets are real, enforced
data: five JSON manifests under V4/perf/ are validated against measured
profiler output by a 688-line Node gate
(apps/v4/scripts/src/v4-performance-budget-check.mjs) wired into a GitHub
Actions workflow that fails the build on a frame, headroom, or memory
regression.
The honest caveats are equally specific. Observability is spec-level, not
wired in engine. V4's architecture names Grafana, Prometheus, OpenTelemetry,
and Sentry, but — unlike V2, which binds every service to @oshun/metrics /
@oshun/tracing through an in-engine FV2ServiceObservabilityBinding — V4 has
no equivalent binding object on disk, and its Rust services do not yet
declare those exporters. The golden-replay regression corpus the testing
section describes (V4Tests/GoldenReplays/) is not present as a directory;
the replay test that does exist covers the replay viewer and camera presets, not
a frame-exact divergence corpus. And the frame/memory numbers are a contract a
CI gate measures against, not a runtime guarantee — the correct framing for a
budget.
Telemetry#
The client subsystem (V4Telemetry, real C++)#
UV4TelemetrySubsystem lives in
V4/ue/Source/V4Telemetry/Private/V4TelemetrySubsystem.cpp and is a
GameInstanceSubsystem — one instance per running game, surviving level loads,
which is exactly the lifetime an analytics queue wants. Its module declares only
Core, CoreUObject, Engine, V4Core, HTTP, and Json
(V4Telemetry.Build.cs), so it has no dependency on the deterministic gameplay
modules and cannot perturb them.
The event model is three USTRUCTs in V4TelemetryTypes.h:
FV4TelemetryEvent (an EventName, UserId, RegionCode, a sorted
TMap<FString,FString> of attributes, and a UTC timestamp), FV4TelemetryBatch
(a BatchId, an event array, and the endpoint/cert-pin carried with it), and
FV4TelemetryEventSchema (a name plus its RequiredAttributes and
bModeSpecific/bMonetization flags). BuildDefaultEventSchema() returns the
canonical taxonomy as data: the base lifecycle events match_start,
match_end, player_move, player_fire, player_death, objective_complete;
the per-cell mode_specific.* family (tactical.breach,
stealth.disguise_changed, stealth.hitman_kill, tactics.showdown_commit,
arpg.parry_success, rts.build, arcade.powerup_pickup); and the
monetization.* family (purchase, ledger_entry, refund). Typed factories
— MakeMatchStartEvent, MakeModeSpecificEvent, MakeMonetizationEvent, and
the rest — build well-formed events and normalize the cell/verb components to
lower-snake-case, so a Hitman kill becomes mode_specific.stealth.hitman_kill
carrying map_id, room_id, method_id, target_id, and season_id.
Every event passes ValidateEventAgainstSchema() before it is accepted: an
unknown event name or a missing required attribute is rejected with a specific
error string (the wildcard rows mode_specific.* and monetization.* let new
verbs validate against the family contract). That validation runs inside
EnqueueEvent(), after SanitizeEvent() assigns a sequence-stamped EventId,
uppercases the region, and stamps CreatedAtUtc — so malformed or opted-out
events never enter the buffer.
Batched sender, offline queue, and privacy#
The sender mirrors the architecture's "256 KB buffer, flush every 30 s" spec as
real defaults in FV4TelemetrySenderConfig (MaxBufferBytes = 256 * 1024,
FlushIntervalSeconds = 30.0, default MaxBatchSize = 32). Two predicate
methods drive flushing: ShouldFlushForBuffer() (queued bytes ≥ the buffer cap)
and ShouldFlushByTime(NowUtc) (elapsed since last flush ≥ the interval).
BuildNextBatch() drains up to the batch limit into an FV4TelemetryBatch and
recomputes the buffered byte count via EstimateEventSizeBytes(), which
measures the actual UTF-8 serialized size rather than guessing.
Offline handling is genuine, not a comment. SetOnline(false) routes new events
into a separate OfflineEvents array; PersistOfflineQueue() serializes them
to a timestamped JSON file under the configured directory (or
ProjectSavedDir()/Telemetry/OfflineQueue), and LoadOfflineQueueFromDisk()
reads them back, re-parsing each event object. SetOnline(true) calls
RetryOfflineEvents(), which appends the offline backlog into the live queue.
The privacy layer is the part that matters for compliance:
SetRegionOptOut() records an FV4TelemetryPrivacyRule per normalized region
and purges already- queued events from that region from both the live and
offline buffers, while IsRegionOptedOut() blocks future enqueues. Transport is
fail-closed: ConfigureTransport() rejects an endpoint that is empty, non-HTTPS
(when bRequireHttps), or missing a certificate pin (when
bRequireCertificatePin), and IsCertificatePinAccepted() does a
case-insensitive compare against the configured SHA-256 pin.
The automation test
(V4/ue/Source/V4Tests/Private/V4TelemetryTests/TelemetrySpec.cpp) is not a
shape check — it asserts the behavior: an http:// endpoint is rejected and an
https:// one with a pin is accepted; the pin compare is case-insensitive;
schema validation rejects an event with match_id removed; the Hitman and RTS
mode-specific names are namespaced correctly; a flush produces a two-event batch
and drains the queue; an offline event persists to disk, reloads, and moves to
the retry queue; and a GDPR/CCPA opt-out both rejects new EU events and purges
the existing US-CA queue. It would fail against a stubbed subsystem.
Ingest: telemetry-ingest (real Rust → ClickHouse)#
The backend half is apps/v4/telemetry-ingest/src/lib.rs, an Axum router
mounting POST /v1/telemetry/batch. ClickHouseHttpWriter::from_env() reads
V4_CLICKHOUSE_HTTP_ADDR; build_clickhouse_insert_batch() serializes each
TelemetryEventRow to a newline-delimited JSON row and rejects a batch over
max_batch_bytes (256 KB — the same contract as the client), and
write_batch() issues the real wire request
POST /?query=INSERT INTO v4_telemetry.events FORMAT JSONEachRow with the rows
as the body. TelemetryBatcher is the size/interval buffer in front of it,
flushing on the same should_flush() predicate (bytes ≥ cap or elapsed ≥
interval). The honesty seam is explicit in accept_batch(): with a configured
writer it returns written: true; with none it returns written: false and
write_error: "not_configured" — a fail-loud report, never a fabricated
success. The Rust tests prove the wire format against a fake ClickHouse TCP
server (writer_performs_a_real_batched_insert_over_http asserts the exact
percent-encoded INSERT line) and assert the unconfigured path
(route_reports_honestly_when_unconfigured). From ClickHouse, the architecture
feeds Grafana dashboards per cell/region/mode and the balance dashboards that
drive automated alerts.
Observability#
V4's Observability, Evaluation, Release Gates section names a conventional
stack — Grafana dashboards, Prometheus metrics, OpenTelemetry
traces, and Sentry for client and server error reporting — with
ClickHouse as the analytics store behind the telemetry pipeline above. That
is the operational intent, and it is the right shape. The honest status,
verified against the tree, is that this layer is specified, not yet wired in
code: there is no in-engine observability binding in V4/ue/Source (a grep
for an ObservabilityBinding analogue returns nothing), and the Rust services
under apps/v4 emit through Axum and serde without declaring Prometheus or
OTel exporters today. This is a real gap relative to V2, where the observability
libraries are a hard contract on every service.
The shared platform libraries those names refer to do exist and are real:
@oshun/metrics (libs/shared/metrics) is a Prometheus/prom-client registry
with an OpenTelemetry bridge, and @oshun/tracing (libs/shared/tracing) wraps
OpenTelemetry with AWS X-Ray propagation and Hono middleware. They are consumed
by the TypeScript service domains elsewhere in the monorepo; V4's Rust + UE5
stack does not link them, so the V4 observability story is "point the fleet at
the shared platform conventions," documented for the platform layer at
../../platform/overview.html, and implemented as
infrastructure config rather than as a V4 runtime object. Treating that as
finished would be the fabrication this page exists to avoid; it is honestly
backlog.
What V4 does have on disk for live operation is the measured artifact that
backs the perf dashboards — V4/perf/performance-regression-baseline.json,
captured (per its captureSource) from "Unreal Insights CSV + RHI memory
export" over a 15-minute window — and the telemetry stream itself, which is the
data source a Grafana balance dashboard would render. The pieces that turn that
data into a watched production system are the open work.
Performance budgets#
This is the most thoroughly-implemented part of the chapter, because it is a CI gate rather than a runtime promise — and CI gates can be made real and run.
The budget contract#
V4/perf/performance-budgets.json encodes the per-cell frame split and the
per-platform memory envelope as data. Each of the five launch cells
(TacticalFPS, RTST, ARPG, RTS, Arcade2D) carries a cellFrameProfiles
entry with cpuMs, gpuMs, audioMs, networkMs, and headroomMs; the
flagship is PS5 Tactical FPS — CPU 12 ms and GPU 13 ms (the two run on parallel
threads inside the 16.6 ms frame, not summed), audio 0.6 ms, network 0.4 ms
(async, non-blocking), and a deliberate 2 ms of live-service headroom. Ten
platform budgets (PC Ultra 4K, PS5 Performance/Quality/Pro, XSS, Switch 2,
macOS, Steam Deck, iPad, Android) each give every cell a targetFps, a
frameMsBudget, and a resolution, plus a memoryBudgetMb block. PS5's memory
envelope is the canonical one: 14 GB available, 6 GB game, 5 GB streaming (3 GB
reserved to the OS), with an 8 GB active-GPU target on the unified memory.
The gate that enforces it#
apps/v4/scripts/src/v4-performance-budget-check.mjs is the real enforcement,
and it does four jobs. First, it validates the budget's internal coherence:
exactly five launch cells, at least ten platforms, every cell with a positive
frame profile, headroom ≥ 2 ms, frameMsBudget ≤ 1000 / targetFps, and
gameMaxMb + streamingMaxMb ≤ systemRamAvailableMb. Second, it checks
measured output against budget: for every platform/cell measurement in
performance-regression-baseline.json it asserts averageFrameMs and
p95FrameMs are within the frame budget, that
cpuMs/gpuMs/audioMs/networkMs do not exceed the profiled cell budget,
that headroomMs has not regressed, and that
gameUsedMb/streamingResidentMb/activeGpuMb stay under the platform memory
caps — any breach pushes a typed issue and the process exits non-zero. Third,
it validates engineering polish (next section). Fourth, it runs a
doc-coherence gate — the features doc, this architecture document, and
V4_TODOS.md must each still contain the polish tokens (e.g.
pso-delta-report.json, BuildAdaptiveMissionPreloadQueue), so the prose
cannot silently drift from the manifests. The GitHub Actions workflow
.github/workflows/v4-gates.yml runs the check on any PR touching V4/perf/**
or the script, and on push to main, with read-only contents and in-flight-run
cancellation.
Engineering polish: PSO, shader warmth, and cold load#
V4/perf/engineering-polish.json plus its two satellite reports extend the gate
into the launch-stutter and load-time problems that sink an otherwise-shipped
build. PSO regression: pso-delta-report.json tracks per-patch
added/removed pipeline-state-object entries against four required caches
(PSO.FirstLaunch.Core/Cells, PSO.Patch.Current/Delta); the gate fails if any
required entry is missing or if the captured shader-compile stall exceeds the
0.5 ms budget (the report records maxObservedStallMs: 0.0 with a .utrace
capture replay). Shader-warm coverage: the v4.shader_prewarm_coverage
telemetry event must show ≥ 98% of shipping shader permutations hit during
idle-warm on PS5 and XSX (the manifest measures 98.87% and 98.57%). Adaptive
preload: the gate requires the runtime
UV4ShaderWarmupSubsystem::BuildAdaptiveMissionPreloadQueue driven by the AI
Director's mission-intent predictor, with mission-preload-profile.json proving
each cell hits its mission-select p95 (≤ 1500 ms), cold-load reduction (≥ 24%),
and prediction hit-rate (≥ 75%) targets. Cold-load budgets are per-platform
and gated with measured p95s: PS5 9 s (7.8 measured), XSX 9 s (7.9), XSS 14 s
(11.6), Switch 2 22 s (18.4), Steam Deck 12 s. Every one of these is a number
the gate would fail on if the measured artifact regressed past it.
Release gates around the perf gate#
The architecture composes the perf gate with the rest of the release flow:
PR-gate CI (lint, unit, smoke automation, asset linter, dep check), nightly
Gauntlet feel-tests for all six cells, a pre-merge golden-replay regression, and
a full pre-release automation pass with the launch readiness gate's hard targets
(e.g. matchmaking p99 ≤ 35 s at 5× launch concurrency). The honest status of the
test surface: V4/ue/Source/V4Tests holds 90 *Spec.cpp suites across 67
directories — real and broad — but the golden-replay corpus directory the
prose references does not exist on disk yet; what is implemented is the
replay-viewer spec (V4ModeReplayTests/ReplayModeSpec.cpp, per-player PIP
tracks and per-cell camera presets), not a frame-exact divergence corpus. The
performance-budget workflow is the release gate in this chapter that is
genuinely runnable today.
Where this connects#
- The data this measures: the cook and content pipeline that produces the cooked builds and the balance-data exports the dashboards read — ./build-data-content.md.
- The compliance side of telemetry: PII handling, region opt-out regulation (the GDPR/CCPA rules the subsystem enforces), and the launch readiness gate — ./security-compliance-launch.md.
- The shared platform observability libraries (
@oshun/metrics,@oshun/tracing) that V4's spec points its fleet at — ../../platform/overview.html.