# Telemetry, Observability & Performance

V4 ships six genuinely different games — a Rainbow-Six breach, a Hitman sandbox,
a Commandos squad-tactics puzzle, a Black-Myth boss duel, an Age-of-Empires
ladder, and a couch Contra run — onto ten performance targets that range from PC
Ultra at 4K down to an Android phone. Two questions decide whether that
catalogue is shippable and operable: _what is the live game telling us?_ and _is
every cell still inside its frame-time and memory contract on every platform?_
This page is about the three systems that answer them — **telemetry** (the event
stream the client emits and the backend ingests), **observability** (how that
stream and the service fleet are watched in production), and **performance
budgets** (the per-cell, per-platform contract a CI gate holds the build to).
The honest theme, the same one the rest of V4 follows, is that some of this is
real C++/Rust on disk with passing tests, and some is specified policy that a
manifest and a gate enforce rather than a runtime object — and this page draws
that line explicitly so you know which is which. The hub for the V4 architecture
set is [../V4_ARCHITECTURE.md](../V4_ARCHITECTURE.md).

## What ships, honestly

The **client telemetry subsystem is real**: `V4/ue/Source/V4Telemetry` is a
fully-implemented `UGameInstanceSubsystem` (`UV4TelemetrySubsystem`, ~770 lines
of `.cpp`) with schema validation, a byte-bounded batched sender, an offline
disk queue, region opt-out, and an HTTPS-plus-certificate-pin transport policy —
exercised by a real automation test,
`V4.Telemetry.Runtime.QueuePrivacyTransport`. The **ingest path is real Rust**:
`apps/v4/telemetry-ingest` writes batched `INSERT … FORMAT JSONEachRow` rows to
ClickHouse over HTTP and reports `not_configured` honestly when no endpoint is
set — no fabricated success. The **performance budgets are real, enforced
data**: five JSON manifests under `V4/perf/` are validated against measured
profiler output by a 688-line Node gate
(`apps/v4/scripts/src/v4-performance-budget-check.mjs`) wired into a GitHub
Actions workflow that fails the build on a frame, headroom, or memory
regression.

The honest caveats are equally specific. **Observability is spec-level, not
wired in engine.** V4's architecture names Grafana, Prometheus, OpenTelemetry,
and Sentry, but — unlike V2, which binds every service to `@oshun/metrics` /
`@oshun/tracing` through an in-engine `FV2ServiceObservabilityBinding` — V4 has
**no** equivalent binding object on disk, and its Rust services do not yet
declare those exporters. The **golden-replay regression corpus** the testing
section describes (`V4Tests/GoldenReplays/`) is **not present as a directory**;
the replay test that does exist covers the replay viewer and camera presets, not
a frame-exact divergence corpus. And the frame/memory numbers are a **contract a
CI gate measures against**, not a runtime guarantee — the correct framing for a
budget.

## Telemetry

### The client subsystem (`V4Telemetry`, real C++)

`UV4TelemetrySubsystem` lives in
`V4/ue/Source/V4Telemetry/Private/V4TelemetrySubsystem.cpp` and is a
`GameInstanceSubsystem` — one instance per running game, surviving level loads,
which is exactly the lifetime an analytics queue wants. Its module declares only
`Core`, `CoreUObject`, `Engine`, `V4Core`, `HTTP`, and `Json`
(`V4Telemetry.Build.cs`), so it has no dependency on the deterministic gameplay
modules and cannot perturb them.

The **event model** is three `USTRUCT`s in `V4TelemetryTypes.h`:
`FV4TelemetryEvent` (an `EventName`, `UserId`, `RegionCode`, a sorted
`TMap<FString,FString>` of attributes, and a UTC timestamp), `FV4TelemetryBatch`
(a `BatchId`, an event array, and the endpoint/cert-pin carried with it), and
`FV4TelemetryEventSchema` (a name plus its `RequiredAttributes` and
`bModeSpecific`/`bMonetization` flags). `BuildDefaultEventSchema()` returns the
canonical taxonomy as data: the base lifecycle events `match_start`,
`match_end`, `player_move`, `player_fire`, `player_death`, `objective_complete`;
the per-cell `mode_specific.*` family (`tactical.breach`,
`stealth.disguise_changed`, `stealth.hitman_kill`, `tactics.showdown_commit`,
`arpg.parry_success`, `rts.build`, `arcade.powerup_pickup`); and the
`monetization.*` family (`purchase`, `ledger_entry`, `refund`). Typed factories
— `MakeMatchStartEvent`, `MakeModeSpecificEvent`, `MakeMonetizationEvent`, and
the rest — build well-formed events and normalize the cell/verb components to
lower-snake-case, so a Hitman kill becomes `mode_specific.stealth.hitman_kill`
carrying `map_id`, `room_id`, `method_id`, `target_id`, and `season_id`.

Every event passes `ValidateEventAgainstSchema()` before it is accepted: an
unknown event name or a missing required attribute is rejected with a specific
error string (the wildcard rows `mode_specific.*` and `monetization.*` let new
verbs validate against the family contract). That validation runs inside
`EnqueueEvent()`, after `SanitizeEvent()` assigns a sequence-stamped `EventId`,
uppercases the region, and stamps `CreatedAtUtc` — so malformed or opted-out
events never enter the buffer.

### Batched sender, offline queue, and privacy

The sender mirrors the architecture's "256 KB buffer, flush every 30 s" spec as
real defaults in `FV4TelemetrySenderConfig` (`MaxBufferBytes = 256 * 1024`,
`FlushIntervalSeconds = 30.0`, default `MaxBatchSize = 32`). Two predicate
methods drive flushing: `ShouldFlushForBuffer()` (queued bytes ≥ the buffer cap)
and `ShouldFlushByTime(NowUtc)` (elapsed since last flush ≥ the interval).
`BuildNextBatch()` drains up to the batch limit into an `FV4TelemetryBatch` and
recomputes the buffered byte count via `EstimateEventSizeBytes()`, which
measures the actual UTF-8 serialized size rather than guessing.

Offline handling is genuine, not a comment. `SetOnline(false)` routes new events
into a separate `OfflineEvents` array; `PersistOfflineQueue()` serializes them
to a timestamped JSON file under the configured directory (or
`ProjectSavedDir()/Telemetry/OfflineQueue`), and `LoadOfflineQueueFromDisk()`
reads them back, re-parsing each event object. `SetOnline(true)` calls
`RetryOfflineEvents()`, which appends the offline backlog into the live queue.
The **privacy layer** is the part that matters for compliance:
`SetRegionOptOut()` records an `FV4TelemetryPrivacyRule` per normalized region
and **purges already- queued events** from that region from both the live and
offline buffers, while `IsRegionOptedOut()` blocks future enqueues. Transport is
fail-closed: `ConfigureTransport()` rejects an endpoint that is empty, non-HTTPS
(when `bRequireHttps`), or missing a certificate pin (when
`bRequireCertificatePin`), and `IsCertificatePinAccepted()` does a
case-insensitive compare against the configured SHA-256 pin.

The automation test
(`V4/ue/Source/V4Tests/Private/V4TelemetryTests/TelemetrySpec.cpp`) is not a
shape check — it asserts the behavior: an `http://` endpoint is rejected and an
`https://` one with a pin is accepted; the pin compare is case-insensitive;
schema validation rejects an event with `match_id` removed; the Hitman and RTS
mode-specific names are namespaced correctly; a flush produces a two-event batch
and drains the queue; an offline event persists to disk, reloads, and moves to
the retry queue; and a GDPR/CCPA opt-out both rejects new EU events and purges
the existing US-CA queue. It would fail against a stubbed subsystem.

### Ingest: `telemetry-ingest` (real Rust → ClickHouse)

The backend half is `apps/v4/telemetry-ingest/src/lib.rs`, an Axum router
mounting `POST /v1/telemetry/batch`. `ClickHouseHttpWriter::from_env()` reads
`V4_CLICKHOUSE_HTTP_ADDR`; `build_clickhouse_insert_batch()` serializes each
`TelemetryEventRow` to a newline-delimited JSON row and rejects a batch over
`max_batch_bytes` (256 KB — the same contract as the client), and
`write_batch()` issues the real wire request
`POST /?query=INSERT INTO v4_telemetry.events FORMAT JSONEachRow` with the rows
as the body. `TelemetryBatcher` is the size/interval buffer in front of it,
flushing on the same `should_flush()` predicate (bytes ≥ cap **or** elapsed ≥
interval). The honesty seam is explicit in `accept_batch()`: with a configured
writer it returns `written: true`; with none it returns `written: false` and
`write_error: "not_configured"` — a fail-loud report, never a fabricated
success. The Rust tests prove the wire format against a fake ClickHouse TCP
server (`writer_performs_a_real_batched_insert_over_http` asserts the exact
percent-encoded `INSERT` line) and assert the unconfigured path
(`route_reports_honestly_when_unconfigured`). From ClickHouse, the architecture
feeds Grafana dashboards per cell/region/mode and the balance dashboards that
drive automated alerts.

```mermaid
flowchart LR
  subgraph Client["UE5 client · V4Telemetry (real C++)"]
    EV["typed events<br/>MakeMatchStart / MakeModeSpecific"] --> VAL["ValidateEventAgainstSchema"]
    VAL --> Q["EnqueueEvent → byte buffer<br/>256 KB / 30 s"]
    Q -->|opt-out purge| PRIV["FV4TelemetryPrivacyRule"]
    Q --> FB["FlushNextBatch<br/>HTTPS + cert pin"]
    Q -.offline.-> DISK["PersistOfflineQueue (disk)"]
  end
  FB --> ING["telemetry-ingest (Rust/Axum)<br/>/v1/telemetry/batch"]
  ING -->|configured| CH[("ClickHouse<br/>v4_telemetry.events")]
  ING -.unconfigured.-> NC["written:false · not_configured"]
  CH --> GRAF["Grafana · balance dashboards (spec)"]
  subgraph Perf["Performance gate (real CI)"]
    BUD["V4/perf/*.json budgets"] --> CHK["v4-performance-budget-check.mjs"]
    BASE["regression-baseline.json (measured)"] --> CHK
    CHK -->|frame/headroom/memory regression| FAIL["fail the build"]
  end
```

## Observability

V4's `Observability, Evaluation, Release Gates` section names a conventional
stack — **Grafana** dashboards, **Prometheus** metrics, **OpenTelemetry**
traces, and **Sentry** for client and server error reporting — with
**ClickHouse** as the analytics store behind the telemetry pipeline above. That
is the operational intent, and it is the right shape. The honest status,
verified against the tree, is that this layer is **specified, not yet wired in
code**: there is no in-engine observability binding in `V4/ue/Source` (a grep
for an `ObservabilityBinding` analogue returns nothing), and the Rust services
under `apps/v4` emit through Axum and `serde` without declaring Prometheus or
OTel exporters today. This is a real gap relative to V2, where the observability
libraries are a hard contract on every service.

The shared platform libraries those names refer to **do exist and are real**:
`@oshun/metrics` (`libs/shared/metrics`) is a Prometheus/`prom-client` registry
with an OpenTelemetry bridge, and `@oshun/tracing` (`libs/shared/tracing`) wraps
OpenTelemetry with AWS X-Ray propagation and Hono middleware. They are consumed
by the TypeScript service domains elsewhere in the monorepo; V4's Rust + UE5
stack does not link them, so the V4 observability story is "point the fleet at
the shared platform conventions," documented for the platform layer at
[../../platform/overview.html](../../platform/overview.html), and implemented as
infrastructure config rather than as a V4 runtime object. Treating that as
finished would be the fabrication this page exists to avoid; it is honestly
backlog.

What V4 _does_ have on disk for live operation is the **measured artifact** that
backs the perf dashboards — `V4/perf/performance-regression-baseline.json`,
captured (per its `captureSource`) from "Unreal Insights CSV + RHI memory
export" over a 15-minute window — and the telemetry stream itself, which is the
data source a Grafana balance dashboard would render. The pieces that turn that
data into a watched production system are the open work.

## Performance budgets

This is the most thoroughly-implemented part of the chapter, because it is a CI
gate rather than a runtime promise — and CI gates can be made real and run.

### The budget contract

`V4/perf/performance-budgets.json` encodes the per-cell frame split and the
per-platform memory envelope as data. Each of the five launch cells
(`TacticalFPS`, `RTST`, `ARPG`, `RTS`, `Arcade2D`) carries a `cellFrameProfiles`
entry with `cpuMs`, `gpuMs`, `audioMs`, `networkMs`, and `headroomMs`; the
flagship is PS5 Tactical FPS — CPU 12 ms and GPU 13 ms (the two run on parallel
threads inside the 16.6 ms frame, not summed), audio 0.6 ms, network 0.4 ms
(async, non-blocking), and a deliberate 2 ms of live-service headroom. Ten
platform budgets (PC Ultra 4K, PS5 Performance/Quality/Pro, XSS, Switch 2,
macOS, Steam Deck, iPad, Android) each give every cell a `targetFps`, a
`frameMsBudget`, and a resolution, plus a `memoryBudgetMb` block. PS5's memory
envelope is the canonical one: 14 GB available, 6 GB game, 5 GB streaming (3 GB
reserved to the OS), with an 8 GB active-GPU target on the unified memory.

### The gate that enforces it

`apps/v4/scripts/src/v4-performance-budget-check.mjs` is the real enforcement,
and it does four jobs. First, it **validates the budget's internal coherence**:
exactly five launch cells, at least ten platforms, every cell with a positive
frame profile, headroom ≥ 2 ms, `frameMsBudget ≤ 1000 / targetFps`, and
`gameMaxMb + streamingMaxMb ≤ systemRamAvailableMb`. Second, it **checks
measured output against budget**: for every platform/cell measurement in
`performance-regression-baseline.json` it asserts `averageFrameMs` and
`p95FrameMs` are within the frame budget, that
`cpuMs`/`gpuMs`/`audioMs`/`networkMs` do not exceed the profiled cell budget,
that `headroomMs` has not regressed, and that
`gameUsedMb`/`streamingResidentMb`/`activeGpuMb` stay under the platform memory
caps — any breach pushes a typed `issue` and the process exits non-zero. Third,
it **validates engineering polish** (next section). Fourth, it runs a
**doc-coherence gate** — the features doc, this architecture document, and
`V4_TODOS.md` must each still contain the polish tokens (e.g.
`pso-delta-report.json`, `BuildAdaptiveMissionPreloadQueue`), so the prose
cannot silently drift from the manifests. The GitHub Actions workflow
`.github/workflows/v4-gates.yml` runs the check on any PR touching `V4/perf/**`
or the script, and on push to `main`, with read-only contents and in-flight-run
cancellation.

### Engineering polish: PSO, shader warmth, and cold load

`V4/perf/engineering-polish.json` plus its two satellite reports extend the gate
into the launch-stutter and load-time problems that sink an otherwise-shipped
build. **PSO regression**: `pso-delta-report.json` tracks per-patch
added/removed pipeline-state-object entries against four required caches
(`PSO.FirstLaunch.Core/Cells`, `PSO.Patch.Current/Delta`); the gate fails if any
required entry is missing or if the captured shader-compile stall exceeds the
0.5 ms budget (the report records `maxObservedStallMs: 0.0` with a `.utrace`
capture replay). **Shader-warm coverage**: the `v4.shader_prewarm_coverage`
telemetry event must show ≥ 98% of shipping shader permutations hit during
idle-warm on PS5 and XSX (the manifest measures 98.87% and 98.57%). **Adaptive
preload**: the gate requires the runtime
`UV4ShaderWarmupSubsystem::BuildAdaptiveMissionPreloadQueue` driven by the AI
Director's mission-intent predictor, with `mission-preload-profile.json` proving
each cell hits its mission-select p95 (≤ 1500 ms), cold-load reduction (≥ 24%),
and prediction hit-rate (≥ 75%) targets. **Cold-load budgets** are per-platform
and gated with measured p95s: PS5 9 s (7.8 measured), XSX 9 s (7.9), XSS 14 s
(11.6), Switch 2 22 s (18.4), Steam Deck 12 s. Every one of these is a number
the gate would fail on if the measured artifact regressed past it.

### Release gates around the perf gate

The architecture composes the perf gate with the rest of the release flow:
PR-gate CI (lint, unit, smoke automation, asset linter, dep check), nightly
Gauntlet feel-tests for all six cells, a pre-merge golden-replay regression, and
a full pre-release automation pass with the launch readiness gate's hard targets
(e.g. matchmaking p99 ≤ 35 s at 5× launch concurrency). The honest status of the
test surface: `V4/ue/Source/V4Tests` holds 90 `*Spec.cpp` suites across 67
directories — real and broad — but the golden-replay _corpus_ directory the
prose references does not exist on disk yet; what is implemented is the
replay-viewer spec (`V4ModeReplayTests/ReplayModeSpec.cpp`, per-player PIP
tracks and per-cell camera presets), not a frame-exact divergence corpus. The
performance-budget workflow is the release gate in this chapter that is
genuinely runnable today.

## Where this connects

- **The data this measures:** the cook and content pipeline that produces the
  cooked builds and the balance-data exports the dashboards read —
  [./build-data-content.md](./build-data-content.md).
- **The compliance side of telemetry:** PII handling, region opt-out regulation
  (the GDPR/CCPA rules the subsystem enforces), and the launch readiness gate —
  [./security-compliance-launch.md](./security-compliance-launch.md).
- **The shared platform observability libraries** (`@oshun/metrics`,
  `@oshun/tracing`) that V4's spec points its fleet at —
  [../../platform/overview.html](../../platform/overview.html).