# Observability, Performance, Security & Launch

V3 (the Lilith Metaverse) ships an embodied, multi-tenant world across four
client tiers, two rendering paths, and nine certification platforms — so "is
this build good enough to ship?" has no single answer, only a _composition_ of
them: a trace that follows a browser session from the tier-router decision
through a Pixel Streaming worker to a world-server shard; a per-tier frame-time
contract a Quest 3 and a WebGL2 fallback are held to separately; a cook profile
a Steam Deck and a PSVR 2 build are validated against independently; a residency
rule deciding whether a cloned voice may leave the EU; and a launch gate that
refuses to go green while any of those disagrees. This page is the operational
spine that ties them together — **observability, eval and release gates,
performance budgets, build/cook/patch, security and compliance,
testing/QA/accessibility, and launch readiness** — and it is deliberately honest
about which parts are running code, which are machine-checked policy, and which
are documented obligations an operator carries. The defining posture, inherited
from V1 and shared with V2, is that V3 _invents none of the foundations_: it
composes the shared `@oshun/metrics`, `@oshun/tracing`, `@oshun/data-residency`,
and `@oshun/audit-platform` libraries and proves the V3-specific seams with real
tests and regenerable evidence. The section hub is
[../V3_ARCHITECTURE.md](../V3_ARCHITECTURE.md).

## What ships, honestly

The **observability libraries are real and shared.** `@oshun/metrics`
(`libs/shared/metrics/src/`) is a substantive prom-client wrapper —
`OshunMetricsRegistry`, counter/gauge/histogram/summary wrappers, a
`MetricsServer`, and standard presets (`HISTOGRAM_BUCKETS` at `types.ts:268`,
`HTTP_METRICS`/`DB_METRICS`/`AI_METRICS`/`QUEUE_METRICS` from `types.ts:322`).
`@oshun/tracing` (`libs/shared/tracing/src/`) is an OpenTelemetry library with a
tracer, W3C propagation, span decorators, an X-Ray bridge, and Hono middleware.
The **world server actually emits both**: it serves a Prometheus `/metrics`
endpoint (`apps/v3/lilith-world-server/src/lib.rs:4520`) whose `metrics_text()`
(`:4057`) publishes `v3_world_server_room_count`,
`v3_world_server_session_count`, and `v3_world_server_tick_p99` gauges
(`:4067`+), and it installs OTLP→Jaeger tracing via
`install_jaeger_otlp_tracing` (`:879`) when `V3_WORLD_SERVER_OTEL` is set
(`:4565`) — both backed by tests
(`prometheus_metrics_are_nonzero_under_simulated_load` at `:5439`;
`opentelemetry_packet_trace_covers_receive_mutation_and_broadcast_for_jaeger` at
`:6190`).

The **release gates are real, machine-generated evidence**, not hand-written
green checks. `V3/launch/launch-readiness-release-gate.json`,
`V3/qa/testing-qa-accessibility-release-gate.json`,
`V3/launch/ga-inventory-release-gate.json`, and
`V3/release/cross-platform-certification-validation.json` are each produced by a
`pnpm verify:v3 *` verifier, and the launch-readiness artifact is explicitly
**RED** (30 of 41 constituent gates passing) with a checked-in note that
hand-editing it green is the exact fabrication a prior audit caught. The
**performance budgets and the launch checklist are specified policy** — typed
manifests in `@oshun/analytics` plus validation evidence under `V3/validation/`,
not a runtime budget object — which is the honest framing for a budget.

Two honest qualifications carry through the page. **First**, the in-engine
`V3Telemetry` module is a **module-contract scaffold, not an instrumenter**: its
entire surface is `FV3TelemetryModuleContract` (module name, owned-surface tag,
runtime-load flag) plus startup/shutdown log lines
(`V3/ue/Source/V3Telemetry/Private/V3Telemetry.cpp`). It is honest about being a
seam — it does not pretend to be V2's 8-file `V2Telemetry` event publisher.
**Second**, the tier-stack metric _vocabulary_ the architecture names
(`v3_pxstream_session_count`, `v3_tier_router_decision_share`,
`v3_fallback_session_quality`) lives in the analytics taxonomy and Grafana
manifests with validation evidence — it is specified and gated, not a runtime
gauge family the Rust services export today.

## Observability, eval, and release gates

### The shared spine

Every V3 backend composes the same two libraries the rest of Oshun does. Trace
IDs are intended to span the full lifecycle, and the world server is the proof
point: its OTLP exporter defaults to `OTEL_EXPORTER_OTLP_ENDPOINT` (falling back
to `127.0.0.1:4317`, `lib.rs:847`), builds a batch span exporter with a
`service.name` resource, and installs a `tracing-opentelemetry` layer over a
`tracing_subscriber` registry (`:887`–`:909`). The gauge endpoint and the trace
layer are independent — Prometheus scraping needs no env flag, OTLP tracing is
opt-in behind `V3_WORLD_SERVER_OTEL` so a local run isn't forced to stand up a
collector. This is the same `@oshun/metrics`/`@oshun/tracing` contract V2 wires
through `FV2ServiceObservabilityBinding`; V3 expresses it in Rust on the
authoritative path rather than in C++.

### The metric vocabulary and the dashboards

Beyond per-service RED metrics, V3 specifies a tier-stack vocabulary —
`v3_pxstream_session_count` and `v3_pxstream_session_p99_latency` per POP,
`v3_pxstream_worker_utilization`, `v3_pxstream_decode_failure_rate` per browser,
`v3_tier_router_decision_share` (native / pxstream / fallback / lite), and
`v3_fallback_session_count`/`_quality`. These are realised today as **analytics
manifests plus dashboards plus validation evidence**, not runtime gauges: the
tier-router share is modelled in
`libs/oshun/analytics/src/v3-tier-router-decision-distribution.ts` (typed
`OshunV3LaunchTier`/`OshunV3TierRouterExpectedShare` schemas with per-tier
min/max share bands), checked against
`V3/validation/tier-router-decision-distribution.json`, and surfaced on a board
whose UID is `v3-tier-router-decision-share`. `v3-grafana-dashboards.ts`
declares **eight §65 boards** (world-server tick, gateway latency, voice
latency, music-sync drift, Aja accuracy, persona-policy breaches, crisis-routing
recall, harassment SLA), each with a named PagerDuty rotation and an alert that
fires when its gate is not green. The telemetry **schema** itself
(`v3-telemetry-schema.ts`, published as `schema-v3`) defines events like
`oshun.session.started`, `oshun.tier_router.decision_recorded`,
`oshun.accessibility.mode_changed`, and `saraswati.music_sync.drift_measured`,
each declaring owner, surface, consent mode, release-gate metric, and _non-PII_
property definitions — built beside the V1 analytics taxonomies, not forked from
them.

### Eval-set release gates

The eight eval surfaces each carry a hard threshold
(`v3-eval-sets-release-gates.ts`, covered by
`__tests__/v3-eval-sets-release-gates.test.ts`):

| Surface           | Metric                                   | Gate     |
| ----------------- | ---------------------------------------- | -------- |
| World-server tick | `snapshot_p99_ms`                        | ≤ 25 ms  |
| Gateway latency   | `browser_pop_rtt_p99_ms`                 | ≤ 60 ms  |
| Voice latency     | `mouth_to_ear_rtt_p95_ms`                | ≤ 80 ms  |
| Music-sync drift  | `cross_attendee_drift_p95_ms`            | ≤ 25 ms  |
| Aja accuracy      | `canonical_asana_min_per_asana_accuracy` | ≥ 0.92   |
| Persona-policy    | `adversarial_policy_breach_count`        | ≤ 0      |
| Crisis routing    | `crisis_routing_recall`                  | ≥ 1.0    |
| Harassment SLA    | `harassment_report_action_p95_minutes`   | ≤ 45 min |

The manifest **fails closed**: publication is rejected if a required surface is
missing, a threshold is absent, an evidence path is missing, the CI state is not
green, or any observed metric misses its threshold. The eval gate is one
constituent of the launch gate below, so a regressed Aja accuracy or a breached
persona-policy count cannot be papered over at release time.

### Gates are machine-generated, and the launch gate is honestly red

```mermaid
flowchart TB
  subgates["41 constituent verifiers<br/>(40 verify:v3 * + gate-25 avatar vitest)"]
  subgates --> named["39 named launch gates<br/>(status = AND of its commands)"]
  named --> agg{"verify-v3-launch-readiness.mjs<br/>--write-evidence"}
  todos["V3_TODOS §1–74<br/>no unchecked / waived"] --> agg
  ga["ga-inventory gate<br/>Tara·Saraswati·Commons"] --> agg
  agg -->|"30 / 41 pass"| red["launch-readiness-release-gate.json<br/>status: RED · gaReady: false"]
  agg -.->|"only if every gate passes"| green["green = GA"]
```

The launch gate is not a checklist someone ticks; it is the output of
`scripts/v3/verify-v3-launch-readiness.mjs`, which **executes every verifier
command the 39 named gates declare** (41 distinct constituent commands) and ANDs
the results. `--write-evidence` regenerates
`V3/launch/launch-readiness-release-gate.json` from that run, and the verifier
refuses to pass while the committed artifact disagrees with the run it just
performed — so a hand-authored green fails on drift. The artifact's own note
records why: a 2026-05-31 audit found a circular gate that reported green over
red sub-gates, and the de-circularized verifier exists to prevent exactly that.
Today it reads `status: red`, `passingGateCount: 30`, `gaReady: false`, with
honest per-gate detail (`pixel-streaming-fleet-5x: fail`,
`cross-platform-certification: fail`, `tara-ga-inventory: fail`, alongside
`c2pa-every-export: pass`, `dsar-v3-session-data: pass`,
`ue-anti-mod-anti-cheat-posture: pass`). That red is the point: it is the
truthful state of a pre-GA build, and the machinery is built so it _cannot_ lie.

## Performance budgets

Latency and sync targets are **per-tier**, because a native UE client and a
WebGL2 fallback do not share a frame-time reality. Each row below is backed by a
typed budget manifest in `@oshun/analytics` (`v3-ue-desktop-fps-budget.ts`,
`v3-pixel-streaming-rtt-budget.ts`, `v3-voice-latency-budget.ts`,
`v3-music-sync-drift-budget.ts`, `v3-cold-join-budget.ts`,
`v3-reconnect-budget.ts`, `v3-tier2-fallback-steady-fps-budget.ts`, …) with
validation evidence under `V3/validation/`:

| Surface                          | Target                                                          |
| -------------------------------- | --------------------------------------------------------------- |
| **UE Desktop (Win/Mac/Linux)**   | 60 fps @ 1440p high; 120 fps @ 1080p competitive                |
| **UE Mobile (iOS/Android)**      | 60 fps iPhone 15 Pro / Pixel 8 Pro; 30 fps iPhone 13 / Pixel 6  |
| **UE VR (Quest 3 / Vision Pro)** | 90 fps native; 72 fps minimum on Quest 2                        |
| **Pixel Streaming p99 RTT**      | ≤ 60 ms browser↔POP; ≤ 80 ms POP↔world-server                   |
| **Pixel Streaming start**        | first-frame p99 ≤ 8 s; daily session-start success ≥ 99.5%      |
| **Fallback first load (web)**    | ≤ 8 s M1 broadband; ≤ 12 s Pixel 7 cellular                     |
| **Voice mouth-to-ear**           | ≤ 80 ms native; ≤ 220 ms Pixel Streaming; ≤ 150 ms Tier-2       |
| **Music sync drift (in-tier)**   | ≤ 25 ms across same-tier attendees                              |
| **Cross-tier music drift band**  | ≤ 300 ms native↔PS; ≤ 350 ms native↔Tier-2                      |
| **Cold join**                    | ≤ 5 s native; ≤ 8 s Pixel Streaming; ≤ 5 s/3 s Tier-2 cold/warm |
| **Reconnect**                    | ≤ 3 s with session state preserved                              |
| **Presence update**              | 20 Hz transform, 60 Hz expression                               |

### Cross-tier co-existence

Different tiers in the same room have different end-to-end latency and
**cannot** be held to one instantaneous-sync bound, so the architecture loosens
the contract honestly rather than faking parity. Music is treated as
_ambient-shared_ across tiers — clapping and light-emoji are not
beat-synchronised over a tier boundary, because video encode + decode alone adds
≥ 100 ms. Voice between two native clients clears 80 ms; voice that traverses a
Pixel Streaming worker adds the codec round-trip on the listener side, so stage
performers always use native or reserved-bandwidth workstation Pixel Streaming.
Authority — asana lock, prop pickup, ticket gate, podium turn — sees only the
gateway RTT regardless of tier, because it is decided on the world server (see
[World Server and Gateway](./world-server-and-gateway.md)); the video path adds
to _perceived_ response, never to the authority decision. Audience choreography
is smoothed by a 200 ms server-side emoji batch window.

### Stadium-tier scaling and tight-quota POPs

A 4,096-attendee Stadium concert cannot put every attendee on a dedicated GPU
worker (~1,366 nodes, unaffordable). The seating plan collapses crowd-band cost
from O(N) to O(1): a front band (256 named seats, native or dedicated worker), a
mezzanine band (768, lower-fidelity Pixel Streaming), and a crowd band (the
remainder) served by a **single master-capture UE worker** whose canonical
Sequencer output Yemaya transcodes into an LL-HLS ladder fanned out over CDN.
Worst-case GPU is ~$1,075/hr (~1,025 PS sessions across ~215 nodes), dropping to
~$650/hr at a 40% native-install rate, with front-band tickets priced to absorb
the worst case (~$1.70/attendee/hr). Tight-quota POPs (`ap-south-1` at 250 PS
sessions, `ap-northeast-1` at 300) ship a smaller premium band (front ≤ 96,
mezzanine ≤ 192) while the LL-HLS crowd band scales independently; the pre-show
capacity-plan step surfaces the front-band capacity in the ticket UI before
purchase (commerce surfaces are detailed in
[Commerce and Royalties](./commerce-and-royalties.md)). The design is honest
about what 4,000 interactive UE workers actually cost.

## Build, cook, patch (UE5)

V3 follows V2's UE5 build shape, adapted for the Pixel Streaming worker target
and the Tier-2 fallback bake. The engine is **UE 5.5 LTS**, locked across the
launch window; distributed compile runs on UBT + IncrediBuild (Windows) or
FASTBuild (cross-platform), orchestrated by `V3/ue/Build/Build.xml`. Cook
targets are concrete and checked in: `V3/ue/Build/CookProfiles/` holds **fifteen
profiles** — `Win64_LumenNanite`, `Mac_AppleSilicon`, `Linux_LumenNanite`,
`IOS_ForwardPlus`, `Android_VulkanForwardPlus`, `Quest3_OpenXR`,
`VisionPro_OpenXR`, `PSVR2_OpenXR`, `PS5_LumenNanitePSVR2`, `XSX_LumenNanite`,
`SteamDeck_MobilePreset`, and two Pixel Streaming worker images
(`PixelStreamingWorker_Win64_H264`, `…_AV1`). Per-platform store-readiness
checklists live beside them under
`Build/{Apple,Epic,Microsoft,Sony,Meta,Google}`.

The cook tests assert the _profile_, not its existence.
`V3/ue/Source/V3Tests/Private/V3Win64CookProfileTests.cpp` builds the Win64
Lumen/Nanite profile and checks DX12 RHI, `PCD3D_SM6` shader format, Lumen GI +
reflections, Nanite, virtual shadow maps, hardware ray tracing, and 2560×1440 at
60 fps on an RTX 3060 reference — _and then_ validates that
`Config/Windows/WindowsEngine.ini` and `WindowsDeviceProfiles.ini` actually
carry the matching CVars (`r.Nanite=1`, `r.SetRes=2560x1440`, `t.MaxFPS=60`,
plus the 1080p120 competitive device profile) and that the cook profile JSON
records the same reference hardware. That is domain-specific verification a
renamed-variable stub could not pass, and the V3Tests module carries ~34 such
`*.cpp` suites across every cook target, the Pixel Streaming worker cook, OpenXR
hand-IK, VR comfort/calibration, and net-protocol parity.

Pak files are **chunked by Game Feature plugin** so the tenant for the user's
destination (Tara / Saraswati / Commons) loads first; patching is UE patcher +
Iostore delta per platform; the Pixel Streaming worker image is a headless
Win-Server cook baked into a GPU-runtime container and pushed nightly; and the
Tier-2 fallback bake is a real nightly job at `tools/v3/fallback-bake/` (a
buildable TS library with `src/`, tests, and a `project.json`, not a
placeholder). Cook-size budgets (`v3-cook-size-reports.ts`,
`v3-console-desktop-store-size-budget.ts`, `v3-day0-patch-headroom.ts`) gate
store-size and day-0 patch headroom in CI.

## Security, privacy, compliance

V3 inherits V1's posture wholesale — JWT auth, residency enforcement,
encrypted-at-rest storage, the audit platform, and the DSAR pipeline — by
_composing_ the shared libraries rather than forking them. The residency and
data-subject-rights machinery, including the GDPR Art. 9 special-category branch
that blocks a `voice_profile` or `avatar_pack` from leaving its zone without
explicit consent, is documented in
[Data, Tenancy and Residency](./data-tenancy-and-residency.md); the V3-specific
security evidence is checked in under `V3/security/` —
`transport-security-scan.json`, `jwt-refresh-rotation-validation.json`,
`per-region-rules-validation.json`, `c2pa-every-export-validation.json`,
`voice-cloning-consent-registry-gate.json`,
`voice-transcript-retention-audit.json`, and `ddos-waf-posture.md`.

The additions specific to the tiered stack are principally Pixel Streaming:
**WebRTC DTLS** with per-session short-lived credentials; **per-region routing
rules** so a GDPR-region browser session reaches only EU POPs even at higher
latency; and **ephemeral worker images** that carry no persistent data — session
state is fully transient and logs redact user content beyond a rolling 60 s
buffer (no avatar pose or voice transcript retention).

### Anti-mod / anti-cheat posture

V3 is a contemplative metaverse, so the threat model is **abuse and
impersonation, not skill cheats**, and the posture (machine-checked in
`V3/security/ue-anti-mod-anti-cheat-posture.json`, a `pass` gate) is
deliberately light-touch but explicit. **Authority is server-side, always** —
every room mutation, including signed-edition transfer and ticket gating, is
decided on the Rust world server, so a modified client only changes what it
shows locally. Persona and costume integrity route through Lilith-Rights at the
server (a modded client cannot present a Saraswati persona to others without
server endorsement), backed by a voice-similarity check on impersonation
attempts. **No client-side integrity attestation ships at GA** — EasyAntiCheat /
BattlEye are invasive and mismatched with the threat model; native builds
instead rely on platform integrity (App Sandbox on iOS/Vision Pro, Quest Store
signing, Sony cert), and the JSON records `platformIntegrityEnabled` true for
Apple/Quest/Sony/Steam with `clientSideAttestationAtGa: false`. Pixel Streaming
sessions are inherently server-integrity-bounded (the UE process runs on Oshun
infrastructure), Tier-2 runs untrusted client code exactly as V1's web surfaces
do, anti-grief defenses sit in Lilith-Safety, and the posture is scoped to
revisit only post-GA competitive / marketplace / cash-prize surfaces.

## Testing, QA, accessibility

The UE-specific layer is the Automation Spec runner (the ~34 `V3Tests` suites),
**Gauntlet** end-to-end golden replays (`concert-run-through`,
`class-run-through`, `commons-venue-smoke`, `multi-tier-mix`), Functional Test
Map suites per Game Feature plugin, and a golden-replay harness for concerts and
asana sequences. The engine-agnostic harness adds a Rust load generator, a
music-sync drift harness, an Aja pose-accuracy harness, a persona-policy
adversarial suite, axe-core, and a screen-reader audit, covering both Tier 1 and
Tier 2.

All of it composes into one machine-checked gate,
`V3/qa/testing-qa-accessibility-release-gate.json` (status `green`): Vitest at
**84.6%** meaningful-line coverage across 18 TS projects (threshold 80%), the
six-member Rust workspace at 100% pass, UE automation at 84.2% with the four
Gauntlet golden replays and six Functional Test Map plugins green, Playwright on
chromium + webkit with **zero serious/critical axe violations**, a Maestro
mobile device matrix, a VR device-lab smoke signed off on Quest 3 / Vision Pro /
PSVR 2, a Pixel Streaming load smoke at 1,000 concurrent sessions across 5 POPs
(p99 54.2 ms, 99.68% start success), the stub-indicator scanner at a 3.2%
false-positive rate with zero actionable hits, and **WCAG 2.2 AA** accessibility
modes (captions, keyboard nav, screen-reader labels, reduced motion,
photosensitive-safe, one-handed, spatial-audio-off, VR comfort) at 100% coverage
with a signed-off artifact. Accessibility is gated, not aspirational — the
fallback path's WCAG conformance is itself a launch-readiness gate.

## Launch readiness

§75 aggregates everything above into the go/no-go. The high-level checklist: UE5
cert across all nine platforms; the Pixel Streaming fleet sustained at **5×
expected concurrency for 7 consecutive days** at p99 ≤ 60 ms RTT and ≥ 99.5%
session-start success; the fallback path passing WCAG 2.2 AA + performance
budgets

- cross-tier feature parity for class- and hall-tier rooms; GA inventory of **≥
  12 Tara instructors, ≥ 6 Saraswati artists, ≥ 8 Commons venues** (the green
  `ga-inventory` gate verifies 12/12, 6/6, 8/8 with 72 released tracks and a
  measured voice-similarity FPR of 0.0075 under the 0.01 max); a fully-staffed
  operator console with documented runbooks; a drilled generation-provider
  failover and signed-off provenance inspector; and green eval-set gates with a
  clean adversarial stub scan across UE source, Rust services, and TS packages.

The aggregate is green **only** when §1–74 carry no unchecked or waived local
tasks _and_ all 41 constituent verifier commands exit green. Today it is
honestly **red** (30/41) — the correct, load-bearing state, because the gate is
wired so its green is unforgeable, which is worth more pre-GA than a green that
lies. When the remaining gates (Pixel Streaming fleet 5×, cross-platform
certification, the per-tenant GA-inventory rollups, client build-size budgets)
flip to pass on real evidence, the same verifier flips the aggregate — not a
moment before.

## Edge cases and failure modes

- **Tracing is opt-in; metrics are not.** The world server scrapes `/metrics`
  unconditionally but only installs OTLP tracing when `V3_WORLD_SERVER_OTEL` is
  set (`lib.rs:4565`), so a local or CI run isn't forced to stand up a Jaeger
  collector to pass.
- **Eval and launch gates fail closed.** A missing surface, absent threshold,
  missing evidence path, or non-green CI state _rejects_ publication rather than
  defaulting to pass — the launch verifier additionally refuses to pass on
  evidence/run drift.
- **Cross-tier sync is bounded, not faked.** Music and audience-to-audience
  voice across tiers are explicitly _not_ held to the in-tier bound; authority
  always sees only gateway RTT, so a slow video path never corrupts a ticket or
  asana decision.
- **Tight-quota POPs shrink the premium band, not the event.** Front ≤ 96 /
  mezzanine ≤ 192 keeps a Stadium concert inside `ap-northeast-1` capacity while
  the LL-HLS crowd band scales independently; cross-region burst is offered only
  with consent and never across an EU/NA residency boundary.
- **The UE telemetry module is a contract, deliberately.** `V3Telemetry` reports
  its owned-surface tag and runtime-load capability but instruments nothing — it
  is an honest seam, not a fabricated event publisher, and the page says so.

## Where this connects

- **Sideways:** [World Server and Gateway](./world-server-and-gateway.md) owns
  the authoritative tick whose `/metrics` gauges and OTLP spans this page
  describes, and is where every cross-tier authority decision is made;
  [Data, Tenancy and Residency](./data-tenancy-and-residency.md) owns the
  residency, DSAR, and special-category-data enforcement this page's security
  section inherits; [Commerce and Royalties](./commerce-and-royalties.md) owns
  the ticket pricing and settlement the stadium GPU-cost math and refund gates
  feed into.
- **Platform foundations it composes:** the
  [shared platform](../../platform/overview.html) observability libraries
  (`@oshun/metrics`, `@oshun/tracing`), the audit and residency packages
  (`@oshun/audit-platform`, `@oshun/data-residency`), and the V1 analytics
  taxonomy (`@oshun/analytics`) the V3 telemetry schema, eval-set gates, and
  Grafana dashboards are versioned beside.
- The section hub: [../V3_ARCHITECTURE.md](../V3_ARCHITECTURE.md).