# Observability, Performance, Testing & the Content Pipeline

V5 is one Unreal Engine 5 open-world narrative universe split into five playable
ruleset cells — a 1947-noir **Urban** cell, a Prohibition-era **Period** cell,
an 1899 **Frontier**, a Witcher-flavoured **Hunter** cell, and a hard-SF
**Sci-Fi** cell — stitched together by a cross-cell **Mind Palace**, and
targeting nine launch platforms from a PS5 down to an iPad. So the question this
page answers — _how do you know the build is good enough to ship?_ — has no
single answer, only a composition: a telemetry event that survives batching and
schema-validation at the client edge before it ever reaches ClickHouse; a
per-platform frame-time budget a Switch 2 and a Windows-ultra rig are held to
separately; a feel test that proves an Urban two-protagonist chase swap still
feels like the GTA scene it was modelled on; and a launch gate that refuses to
go green while any of those disagrees. This page is the quality spine that ties
them together — **observability, performance budgets, testing and the eval
gates, the content pipeline, and the launch-readiness gate** — and it is
deliberately honest about which parts are running code, which are
machine-checked policy, and which are documented obligations that no build yet
satisfies. The defining posture, shared with V2 and V3, is that V5 _invents none
of the foundations_ where it can compose them, and that a gate's red is worth
more than a green that lies. The section hub is
[../V5_ARCHITECTURE.md](../V5_ARCHITECTURE.md).

## What ships, honestly

The **in-engine telemetry is real and substantive**, not a contract scaffold.
`V5Telemetry` (`V5/ue/Source/V5Telemetry/`, `V5TelemetrySystems.cpp` ~299 lines,
`V5TelemetryTypes.h` ~220 lines, a 140-line test) implements the actual event
batcher, a TypeBox schema validator, the 32-entry critical-event taxonomy, the
opt-out filter, and the DSAR export as deterministic Blueprint function
libraries. The **testing surface is real**: `V5/ue/Source/V5Tests/` carries a
511-line C++ harness (`V5TestSystems.cpp`) plus a 5,804-line suite file with
**59 `IMPLEMENT_SIMPLE_AUTOMATION_TEST` cases** under the `V5.Tests.*`
namespace, and the **performance budgets are a cross-validating gate** — a
checked-in profile catalog bound to per-platform `.ini` config and proven by a
223-line Python validator and a UE automation test, not a runtime budget object.

Two honest qualifications carry through the page. **First**, V5 does _not_ wire
the shared observability spine into its services the way V2 does: a grep for
`@oshun/metrics` or `@oshun/tracing` across `apps/v5/` returns nothing. Those
libraries are real and substantive in the monorepo
(`libs/shared/metrics/src/registry.ts` is a 65-symbol prom-client wrapper), but
V5's 16 backend services compose `@v5/service-shared` and express observability
through ClickHouse/Grafana telemetry plus contract-declared regional failover,
not a per-service RED-metrics endpoint. **Second**, V5 ships **zero binary
`.uasset` files** (the content is logic and 29 `.uplugin` manifests); the
"content pipeline" below is therefore a real _process and validator_ surface
whose teeth today are 59 manifest tests and 68 Python validators, not authored
art. The **launch-readiness gate is honestly red** — 17 of 17 blocking gates
`pending`, `releaseBlocked: true` — and the machinery is built so that red
cannot be forged.

## Observability and telemetry

### The in-engine batcher

V5's client telemetry is event-batched, and the batching is a real algorithm.
`UV5_Telemetry_EventBatcher::BuildBatches`
(`V5/ue/Source/V5Telemetry/Private/V5TelemetrySystems.cpp:110`) walks a stream
of `FV5TelemetryEvent`s and flushes a batch when any of three bounds would be
crossed — `MaxBatchBytes` (2048), `MaxEvents` (500), or `MaxBatchAgeSeconds` (5)
— tagging each finalized batch with the flush reason it tripped (`max_bytes` /
`max_events` / `max_age`, or `drain` for the trailing remainder) and recording
whether sequence ordering was preserved. These are the exact
`2 KB / 5 s / 500 events` limits the architecture's Telemetry section names,
implemented rather than asserted.

Validation happens at the edge before transport.
`UV5_Telemetry_Schema_TypeBox::ValidateEvent` (`:151`) resolves each event to a
`typebox.v5.telemetry.<event>.v1` schema and rejects it on a missing schema, a
schema-name mismatch, a payload over `MaxPayloadBytes`, or any missing required
field (`event`, `account`, `region`, `sequence`) — returning a typed
`EV5TelemetryValidationStatus` rather than a bare bool. The critical-event
catalog (`BuildCriticalEventHandlers`, `:191`) enumerates **32 handlers** across
session, mission, combat, Mind Palace deduction/accusation, dialogue
paragon/renegade, vehicle, frontier horse-bond, ship burn/flip/dock, alchemy and
sign-cast, workshop publish/subscribe, online matchmaking/leaderboard/replay,
ops crash/anti-cheat, and currency-balance — each carrying a privacy class
(`AccountOpaque` by default; `privacy_consent_change` is `Anonymous`). Privacy
is enforced, not promised: `ShouldCollectEvent` (`:264`) drops everything for an
opted-out account _except_ the essential `privacy_consent_change` event, and
`BuildExport` (`:273`) assembles a redacted, companion-app-verified, 30-day-SLA
DSAR export filtered to the requesting account's own events. The module is pure
client-side logic — it builds, validates, and redacts; the network transport to
ClickHouse lives on the services side.

### The ingest contract and the shared spine

The server end is the `telemetry` service (`apps/v5/telemetry/`), a NestJS
service whose `contract.json` declares one endpoint —
`POST /v5/telemetry/batch`, `requiresJwt`, `supportsOfflineQueue`, `regional` —
with ClickHouse primary storage, Kafka cache, Grafana analytics, capabilities
`batch-limits` / `schema-validation` / `privacy-opt-out` / `clickhouse-export`,
and a `regional-active-active` failover plan (`iad`/`fra`/`sin` →
`pdx`/`dublin`/`syd`, `dataResidencyPinned: true`). `service.ts` composes
`@v5/service-shared`'s `handleRealRequest` rather than a bespoke handler. This
is the honest shape of V5 observability today: a schema-validating,
residency-pinned ingest contract feeding per-cell Grafana dashboards — not the
per-service Prometheus `/metrics` family that V3's Rust world server exports or
the `@oshun/metrics`/`@oshun/tracing` binding V2 mandates in every service. The
shared spine exists in the platform; V5 has not yet adopted it in-service, and
the architecture's reference to a `apps/v5/telemetry/schema/` directory is a
specification — that path is not populated in-repo.

## Performance budgets

### The platform profile catalog

The per-frame budget is per-platform and explicit, because a Switch 2 and a
Windows-ultra rig do not share a frame-time reality. The architecture's budget
table (PS5 33.3/16.6 ms, PS5 Pro down to 8.3 ms, Switch 2 with a fatter 28 ms
GPU slice, Steam Deck at 33.3 ms, eleven rows in all) is bound to a
launch-locked catalog at
`V5/ue/Content/V5Performance/Data/performance_platform_profiles.json`
(`schemaVersion: 1`, `contentLock: "launch-v5"`): **11 budget profiles across 9
platform groups, with 11 release-blocking regression suites**. Each profile
carries its upscaler (TSR/MetalFX), GI and Nanite strategy, per-mode targets
(`targetFps`, `frameBudgetMs`, `gpuBudgetMs`, `cpuBudgetMs`, a
dynamic-resolution range, and a `profiledFrameP95Ms`), scalability knobs, a
`profilingEvidence` block (a `.utrace` artifact, ≥10 capture-minutes, ≥5
scenarios), and a release-blocking `regression` entry.

### The cross-validating gate

`V5/tools/performance/validate-performance-budgets.py` is the gate, and it does
domain-specific verification a renamed-variable stub could not pass. It checks
the profile inventory against an explicit `EXPECTED_PROFILE_IDS` set and the
exact `(targetFps, frameBudgetMs, gpuBudgetMs, cpuBudgetMs)` tuple for every
profile/mode against an `EXPECTED_BUDGETS` table; it asserts the special
per-platform flags (`switch-2` reduced-Lumen + SDF GI _and_ Hard-Vacuum cook
excluded, `macos-apple-silicon` MetalFX, `steam-deck` Proton-verified,
`ipad-pro-m3` touch-only Mind Palace); it requires
`profiledFrameP95Ms ≤ frameBudgetMs`; and then it **re-parses each platform's
`.ini`** (e.g. `V5/ue/Platforms/PS5/Config/PS5Engine.ini`) and confirms the
config section's `profileId`, memory budget, upscaler, GI/Nanite strategy,
regression-suite id, and every per-mode FPS/budget/resolution match the JSON.
The matching UE test, `FV5PerformanceBudgetCatalogTest`
(`V5.Tests.PerformanceBudgets`, `V5TestsAutomation.cpp:324`), independently
loads the catalog and asserts the 9/11/11 counts, that the profiled p95 stays
within budget, that **every** profile blocks release through its regression
suite, and that the four special-platform flags are set. Two independent readers
— Python in CI and C++ in-engine — must agree with the data, which is how the
budget gets _enforced_ rather than merely documented.

### Arcade cabinet

V5 also cooks a distinct location-based SKU. `V5ArcadeCabinet`
(`V5/ue/Source/V5ArcadeCabinet.Target.cs`) is a four-player local-co-op cabinet
target with its own compile definitions (`V5_ARCADE_CABINET`,
`V5_CABINET_ONLINE_SERVICES=0`), packaged through a dedicated
`ArcadeCabinetEngine.ini`, profiled at 4K60 with a 16.6 ms budget, and validated
by its own manifest test (`FV5ArcadeCabinetBuildTargetManifestTest`). It cooks
only a curated roster (Spec Ops Co-op, Horde Defense, Fleet Co-op, Hunter Card
Game, Training Range) and hides open online/PvP rulesets behind the build-target
gate.

## Testing and the eval gates

### The automation surface

The UE automation layer is `V5Tests` (`V5/ue/Source/V5Tests/`, depending only on
`Core`/`CoreUObject`/`Engine`/`Json`/`V5Core`). The bulk of its **59** suites
are content-as-data manifest validators — they load a plugin's JSON, assert its
schema version, content counts, and stable ids, and check release-gate booleans
(`FV5CrossCellReplayEditorManifestTest`, for instance, verifies 7 supported
cells, 14 source clips, 13 cross-cell splices, and that every clip is
hash-verified, privacy-scrubbed, and rights-cleared). This is honest about what
V5 is today: a logic-and-manifest skeleton whose tests gate the _data_, since
there is no binary content to gate. Three suites, though, exercise the quality
machinery itself.

### Feel tests

The feel-test catalog (`UV5_Test_FeelTestCatalog::BuildFeelTestScenarios`,
`V5TestSystems.cpp:108`) defines **nine reviewer-facing scenarios**, each pinned
to its genre inspiration: Urban heist-chase-swap (GTA) and fish-market brawl
(Sleeping Dogs), Period made-man escort (Mafia) and crime-scene interrogation
(LA Noire), Frontier outlaw-trail ride (Red Dead), monster-hunter contract (The
Witcher), Sci-Fi squad interrupt (Mass Effect) and zero-G duel (The Expanse),
and the cross-cell Mind Palace case file (Sherlock). Every scenario declares a
runtime path, required gameplay systems, and ≥3 blocking success criteria plus
≥3 failure modes. `FV5FeelTestCatalogTest` (`V5.Tests.Feel.Catalog`) asserts
exactly nine, unique ids, and that precisely one — Mind Palace — is the
cross-cell meta-layer. Each scenario is mirrored as a source-controlled
`*.feeltest.json` record under `V5Tests/FeelTests/` so the nightly driver can
assemble its plan without loading content assets, and the C++ catalog and the
nine JSON files are kept in lockstep by the suite.

### Golden-replay determinism

`UV5_Test_GoldenReplayHarness` is the determinism dividend. `BuildInputStream`
(`:331`) emits a deterministic stream — six input events, four expected outputs,
and a `StableFingerprint` computed by `HashCombineFast` over the scenario id,
seed, and every input frame/payload (`BuildFingerprint`, `:71`).
`ValidateReplay` (`:362`) matches expected outputs to actuals within a
**±2-frame tolerance** and an optional numeric tolerance, and returns `Passed`
only when no issue is logged. `FV5GoldenReplayHarnessTest`
(`V5.Tests.GoldenReplay`) proves both directions: a faithful replay passes with
all four outputs matched, and mutating a single actual value to `"false"` flips
the result to `Failed` with a reported issue — a test that would _fail_ against
a stub that always returned success.

### The Gauntlet driver

`UV5_Test_GauntletDriver::BuildNightlyRun` (`:410`) assembles the nightly plan:
**one** parallel worker (conservative for small hosts), the golden-replay
harness plus all nine feel scenarios for ten entries total, each with a real UE
automation command and a per-scenario timeout. `BuildCommandLine` (`:439`) emits
the unattended invocation
(`-NullRHI -NoSound -Unattended -NoSplash -TestExit=... -ExecCmds=... -ReportExportPath=...`),
and `ValidateNightlyRun` (`:447`) refuses a plan that drops a feel scenario,
omits the golden-replay entry, runs more than one worker, or leaves any entry
non-blocking. The architecture's release-gate list (nightly Gauntlet pass, each
cell's feel suite, adversarial stub-scan, >99.5% PS5 crash-free,
memory-within-budget on a 10-hour run, packet-loss tolerance at 5%/20%,
localization and accessibility) composes on top of this driver.

## The content pipeline

### Source control, CI, and where the content actually is

V5 splits its repository in two: the Oshun monorepo (Git) holds all logic,
manifests, and tooling; binary assets live in **Perforce** (Helix Core) under
`V5/ue/Content/`, outside Git's scope, with per-asset checkout locking to avoid
binary merge conflicts. Branching is trunk-based with short-lived feature
branches; CI is **Buildkite** for tests, **Horde** for distributed cook, and
Perforce asset CI. Build/cook/patch mirror V4: UnrealBuildTool + Horde for
distributed compile, per-platform parallel and iterative cook, and chunked
`UnrealPak` iterative patches sized per-cell for partial-download installs. The
deeper cook, packaging, and content-as-data accounting are owned by the sibling
[telemetry, build & data](./telemetry-build-and-data.md) page.

### Validators are the teeth

Because V5 ships **0 `.uasset`** and **29 `.uplugin`**, the content pipeline's
enforcement today is data validation, and it is extensive. `V5/tools/` carries
**68 `test_validate_*.py`** suites plus `validate-v5-docs.py`, which performs
referential integrity across the planning docs and content manifests (every
mission's required NPC exists, every vehicle's required mesh is declared). These
Python validators and the 59 UE manifest tests are two independent checks over
the same JSON, which is what keeps content-as-data honest while the binary
content is unauthored.

### Creator suite

The creator surface is a set of UE-Editor + Slate tools — Mission Editor, Heist
Author, Case Author, Contract Author, Ship Loadout Author, Bestiary Author,
Cinematic Director Mode, the Cross-Cell Replay Editor, and a one-click Workshop
Publisher — each backed by a manifest and a `V5Tests` manifest validator (the
replay editor's, above, is representative).

## The launch-readiness gate (honestly red)

§"Launch Readiness" aggregates everything above into a single go/no-go, and it
is the page's load-bearing example of fail-loud honesty.
`V5/release/launch-readiness-manifest.json` (`schemaVersion: 2`) tracks **17
blocking gates**, and today every one reads `pending` — `passedGates: 0`,
`cellsComplete: 0`, `languagePacksLocked: 0`, `platformsCertPassed: 0`,
`releaseBlocked: true`, `greenBuildId: null`. Its `correctionNote` records why:
a prior revision fabricated a shippable game — 17/17 gates passed, a 99.72%
crash-free rate over 125,000 PS5 sessions, nine platform cert passes, a
completed external accessibility audit — with no packaged build, hardware,
sessions, or external engagement behind any of it.
`V5/REMEDIATION_2026-06-12.md` reset it (and moved the backlog from 905 `[x]` to
875 `[x]`, promoting **zero** marks to done). The note is blunt about the single
real signal: _"the only verified quality signal to date is the one-off
2026-05-31 on-box automation run (298/298 after remediation); no green nightly
pipeline exists."_

The gate cannot be hand-edited green.
`V5/tools/release/validate-launch-readiness.py` (399 lines) **fails closed**: a
gate may claim `passed` only with backing evidence that exists on disk;
`releaseBlocked: false` requires _every_ gate passed _and_ a real
`greenBuildId`; a cert-passed platform needs `memoryWithinBudget` plus an
existing `certEvidencePath`; a "complete" accessibility audit needs a real named
consultant (`"TBD"` is rejected); the crash-free gate needs a cohort ≥100,000
with rate above threshold; and it runs its own regex stub-scan over the runtime
tree for `coming soon|not implemented|todo|fixme|tbd`. It composes the cell,
performance-budget, and live-service validators, so a regression in any of them
blocks launch. The UE test `FV5LaunchReadinessManifestTest`
(`V5.Tests.LaunchReadiness`) mirrors the fail-closed logic in C++ — its own
comment records that it once hardcoded the fabricated 17/17 expectations and now
validates internal consistency instead.

```mermaid
flowchart TB
  subgraph cv["Content & quality validators"]
    cells["validate-cells.py"]
    perf["validate-performance-budgets.py<br/>(11 profiles · .ini binding)"]
    live["validate-live-service.py"]
    stub["runtime stub-scan<br/>(coming soon / todo / tbd)"]
  end
  subgraph ue["V5Tests automation (one-off 298/298)"]
    feel["Feel catalog · 9 scenarios"]
    gr["Golden replay · ±2 frame"]
    gaunt["Gauntlet driver · 1 worker · 10 entries"]
  end
  cv --> gate{"validate-launch-readiness.py<br/>fail-closed"}
  ue --> gate
  manifest["launch-readiness-manifest.json<br/>17 gates · evidence on disk"] --> gate
  gate -->|"0 / 17 passed · no greenBuildId"| red["releaseBlocked: true (today)"]
  gate -.->|"only if every gate passes<br/>on real evidence"| green["green = ship"]
```

## Edge cases and failure modes

- **Telemetry validates and redacts before it leaves the client.** A malformed
  or oversized event is rejected at the edge by `ValidateEvent`; an opted-out
  account emits nothing but the essential consent event; a DSAR export is
  redacted and scoped to the requester. The module fabricates no transport — it
  is honest about being client-side logic feeding the services contract.
- **The performance budget is enforced by two independent readers.** Python in
  CI and C++ in-engine both check the same catalog and `.ini` bindings, so a
  config that drifts from the budget fails the build rather than shipping a
  stutter.
- **The launch gate fails closed, and its red is the point.** Zero of 17 gates
  pass today; the validator refuses a green without on-disk evidence and a real
  build id, and the `correctionNote` documents the exact fabrication the
  fail-closed rewrite exists to prevent.
- **Observability is not yet on the shared spine.** No V5 service declares
  `@oshun/metrics`/`@oshun/tracing`; the honest framing is a ClickHouse/Grafana
  ingest pipeline with residency-pinned failover, not a per-service RED-metrics
  endpoint — a real gap relative to V2/V3, named rather than papered over.

## Where this connects

- **Sideways:** [telemetry, build & data](./telemetry-build-and-data.md) owns
  the cook, arcade-cabinet packaging, and the content-as-data accounting (0
  `.uasset`, 29 `.uplugin`) this page's pipeline and budgets assume;
  [accessibility, security & compliance](./accessibility-security-and-compliance.md)
  owns the WCAG audit, DSAR pipeline, residency, and ratings reviews that
  several launch gates above depend on.
- **Platform foundations it composes:** the
  [shared platform](../../platform/overview.html) observability libraries
  (`@oshun/metrics`, `@oshun/tracing`) — real in the monorepo, and the spine
  V5's services have yet to adopt — beside which the telemetry contract and
  Grafana dashboards are versioned.
- The section hub: [../V5_ARCHITECTURE.md](../V5_ARCHITECTURE.md).