# Observability, Performance, Eval Gates & Launch V7 (Mawu) is a creator republic: a hardened UE5.5 client, a Rust realm fleet, and a platform plane that owns identity, the real-money economy, and child safety — composed so that **untrusted, community-hosted realm logic** runs only inside a capability-sandboxed WASM runtime behind an inviolable trust boundary. For an operations page that means the central question — _is this build good enough to ship?_ — has no single answer, only a composition: a sandbox that must contain a hostile module, a mesh that must hand off authority across a dying node with zero loss, a netcode that must stay playable under packet loss, a payout formula that must settle to the cent, a CSAM hash that must match before any serve, and a launch gate that refuses to go green while any of those is unproven. This page is the operational spine that ties them together — **observability and live operations, performance budgets, build/cook/patch/cert, testing with adversarial eval gates and golden replays, and launch readiness** — and it is deliberately honest about which parts are running code, which are machine-checked policy, and which are still spec inherited from the prior versions. The posture V7 shares with V2 and V3 is that it _invents none of the foundations_: it composes the shared `@oshun/metrics`/`@oshun/tracing` libraries, reuses the V6 realtime gateway and V1 observability stack, and proves the V7-specific seams with real Rust/TS evals and regenerable evidence. The section hub is [../V7_ARCHITECTURE.md](../V7_ARCHITECTURE.md). ## What ships, honestly The **adversarial eval surface is real and large.** `V7/ADVERSARIAL_EVAL_GATES.md` enumerates **29 merge gates**, each owning a seeded hostile fixture, an explicit pass bar, and a concrete CI command — `cargo test` against the `apps/v7`/`libs/v7` workspaces or `pnpm exec vitest` against the TS services. The owning functions exist on disk and are substantial (the `moremi-realm-server` crate alone is >14,000 lines): `run_moremi_ixchel_sandbox_escape_eval`, `run_danu_mesh_handoff_eval`, `run_moremi_golden_replay_ci_gate`, `run_default_realm_netcode_loss_gate`, `run_malware_corpus_eval`, `run_trust_boundary_eval`, and the rest. `.github/workflows/v7-ci.yml` runs an inventory verifier first, then each gate as a named CI step. The **launch gate is real, machine-checked, and honestly negative.** `V7/release/launch-readiness-review.json` reads `decision: "not-launch-ready"`; all eight signoffs carry `status: "fixture-passed"` with the verbatim `evidenceBasis: "in-repo adversarial fixture; not a production measurement"`, and `scripts/v7/verify-launch-readiness.mjs` enforces a **no-fabrication invariant** — it cannot declare `launch-ready` while any signoff rests on fixture evidence (`verify-launch-readiness.mjs:378`–`388`). The staged-rollout record (`V7/release/staged-rollout.json`) has all five stages re-marked `planned` / `pending`, never executed. Three honest qualifications carry through the page. **First**, V7 does **not yet wire** `@oshun/metrics` or `@oshun/tracing` into its Rust services — there is no Prometheus `/metrics` family and no OTLP exporter the way [V3's world server has one](../../V3/architecture/observability-performance-security-and-launch.md). What each service exposes today is a `GET /health` capability contract (`apps/v7/service_contract.rs`). **Second**, the architecture's metric vocabulary (realm population, mesh-node utilization, authority-handoff latency, sandbox-fuel-overrun rate, scan-queue latency, economy inflation) is **specified policy plus in-eval telemetry structs**, not a runtime gauge family. **Third**, there is **no `V7/ue/Build/` cook-profile tree** — unlike V3's fifteen checked-in profiles — because V7 reuses the V5/V6 build pipeline and its UE client is a ~2.2k-LOC procedural skeleton, not a content-complete cook. ## Observability and live operations ### The live service surface that actually exists Every V7 Rust service — Moremi, Danu, Nephthys, Sekhmet, Hera, the Mawu gateway — includes the shared `apps/v7/service_contract.rs` via a `#[path]` module, declares a `const DESCRIPTOR: ServiceDescriptor` (name, owner, port, purpose, capability list), and binds it in `run_service()` through `service_contract::run_health_server(&DESCRIPTOR)`. The handler answers `GET /health` with a JSON capability document — `{"service","owner","version":"v7","status":"ready","port","purpose","capabilities":[…]}` — and 404s everything else (`service_contract.rs:88`). Ports are resolvable per deployment via `V7__PORT` or a fleet-wide `V7_PORT_OFFSET` (`runtime_port`, `:50`). This is an honest seam: a readiness/capability probe a load balancer or the Abundantia browser can poll to flag a degraded realm — **not** a metrics endpoint. The `main.rs` of each service is a one-line delegation to `run_service()`. ### What the spec names, and what is code today The architecture's `Observability and Live Operations` section names the live-ops metric set — realm population and retention, mesh-node utilization and split/merge frequency, authority-handoff latency and failure count, sandbox-fuel-overrun rate, scan-queue latency, payout distribution, per-realm economy inflation, per-client bandwidth — and states that **tracing/logging/SLOs reuse the V1 observability stack** and the gateway (reused from V6's `egbe-realtime-gateway`) exports per-realm health. That is the design; the in-repo reality is narrower and should be read as such. The shared libraries themselves are real and platform-wide (`libs/shared/metrics/src/` — registry, helpers, presets; `libs/shared/tracing/src/` — tracer, W3C propagation, decorators, middleware), but **no V7 service imports them yet** (a grep for `@oshun/metrics`/`@oshun/tracing` across `apps/v7` and `libs/v7` returns zero hits). What V7 _does_ encode today is **domain telemetry inside its eval reports**, not a scrape surface. The §29 cost/fidelity levers emit `DanuDegradationTelemetry { degraded, … }` records when a hot region sheds fidelity (mesh-tier reduction, AI-creation fallback, Crucible reduced coverage), and the eval asserts that all three degradation paths each emit an observable degraded-state record (`danu-mesh-cluster/src/lib.rs:2663`+, asserted at `:3528`). Sekhmet carries `SekhmetTelemetryEvidenceInput` for anti-cheat evidence attachment. Creator analytics (Roblox/UEFN-class DAU/retention/revenue dashboards) is named in the spec but unimplemented. The honest summary: **live ops is a `/health` contract plus in-eval telemetry structs today; the full metric/trace spine is inherited-but-unwired.** ## Performance budgets V7's budgets live in two honestly-different places. The **numeric policy** is stated inline in the architecture and is not a runtime budget object: a 60 fps / 16.7 ms client frame on the reference tier with a 30 fps floor on minimum spec (sustained in a meshed realm of 1000+ via interest-managed culling), authority handoff **≤250 ms p99 with no reconnect**, netcode playable at 5% loss / graceful at 20%, per-client downstream **64–256 kbit/s** in open world, Pheme **≤40 audible streams per listener** after AoI culling with **p95 mouth-to-ear ≤250 ms** at 1000-inhabitant scale, and distribution targets (one-click Collection install p95 ≤120 s for 5 GB on 100 Mbit/s; Sekhmet scan-queue p95 ≤15 min upload-to-verdict). A meaningful **subset is encoded as Rust constants that eval gates assert** — which is stronger than a documented number because a regression fails a test: | Budget | Encoded as | Asserted by | | ------------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------- | | 1000-inhabitant AoI working set | `DANU_AOI_EVAL_INHABITANTS = 1_000` | `area_of_interest_eval_bounds_1000_inhabitant_working_sets` (`danu`) | | Open-world per-client bandwidth | `REALM_DELTA_MIN/MAX_BANDWIDTH_KBPS = 64/256` | netcode-loss gate `within_bandwidth_budget` check (`realm-protocol:2587`) | | Playable-loss threshold | `REALM_INTERPOLATION_EVAL_LOSS_PERCENT = 5` | `netcode_loss_gate_is_playable_at_5_percent_and_graceful_at_20_percent` | | Listen-tier host migration | `LISTEN_HOST_MIGRATION_SLA_MS = 2_000` | `run_listen_host_migration_eval` (`nephthys`) | | Handoff integrity at volume | 200 entities × 10,000 crossings | `mesh_handoff_eval_moves_200_entities_10000_times_without_loss_or_reconnect` | There are **no typed `@oshun/analytics` V7 budget manifests** of the kind V3 ships (`v3-*-budget.ts` + `V3/validation/` evidence); the per-tier frame-time validation V3 has is, for V7, still inline policy plus these constant-backed gates. The honest framing of a budget V2/V3 use applies here too — it is a contract CI and the eval suite hold the build to, sharpened where a constant is load-bearing. ## Build, cook, patch, and cert V7 continues on the UE5.5 LTS / Maya client path and **reuses the V5/V6 build, cook, and patch pipeline** rather than re-inventing it; correspondingly there is no V7-specific `Build/CookProfiles/` tree. The client project `V7/ue/Mawu.uproject` pins `EngineAssociation 5.5`, sets `DisableEnginePluginsByDefault: true` (minimal attack/dependency surface), and enables exactly five plugins — `EnhancedInput`, `CommonUI`, `ModelViewViewModel`, `GeometryScripting`, `GeometryProcessing` — across six runtime/dev modules (`MawuCore`, `MawuRealm`, `MawuBuilder`, `MawuVoice`, `MawuUI`, and the `DeveloperTool` `MawuTests`). Engine config lives in `V7/ue/Config/` (`DefaultEngine.ini`, `DefaultGame.ini`, `DefaultInput.ini`) and a checked-in realm lock at `Config/RealmLocks/dedicated-smoke.lock.json` that the automation suite boots a dedicated realm from. The client has been **built and run on the on-box UE5.5.4 source build** (as the `ueagent` user; the editor refuses root), exercising the `V7.Mawu.*` automation specs green. The distribution mechanics are content-addressed and platform-signed by design (per the Ixchel and Abundantia architecture): every mod artifact's id is the hash of its content plus its full dependency closure (the Nix/lock-file model), giving tamper-evidence, dedup, atomic rollback, and an AOT `.cwasm` cache key; vetted plugins are AOT-compiled to `.cwasm` at upload and production runs with the JIT disabled; client patches are **platform-signed, never operator-distributed**; and console cert constraints gate which sandbox tiers a console realm may grant. Honestly labeled: these are **spec and reused-pipeline**, not V7-specific checked-in cook profiles or a packaged patch artifact. The launch review records the consequence plainly — the UE client is a skeleton, and no realm fleet, packaged build, or store submission exists yet. ## Testing, eval gates, and golden replays ### UE automation: small, real, procedural `V7/ue/Source/MawuTests` holds **12 `V7.Mawu.*` Automation Specs** across two suites — far smaller than V2's 441 or V3's ~34, and the page says so. They are real, not scaffolds: `MawuRealmGeometryTests.cpp` constructs lock files whose primitives carry an `OperationKind`, applies them through `AMawuComposedRealmActor`, and asserts the resulting `UDynamicMesh` triangle/vertex counts and **watertightness** for voxel and CSG solids — proving Geometry Script ops build geometry rather than classify it. `MawuRealmBootstrapTests.cpp` parses the smoke realm lock, composes its six render primitives, boots a dedicated realm through `AMawuGameMode`, validates the exact digest-bound 23-primitive flagship region and a tampered accessibility rejection, drives the builder→realm `OperationKind` mapping, and checks the Pheme voice mix (a matching radio channel routes voice outside proximity range). This is V7's pure-C++ surface; the heavy logic lives in Rust. ### The 29 adversarial gates The eval suite is the load-bearing test layer. Each gate in `V7/ADVERSARIAL_EVAL_GATES.md` "must run in CI, exercise the seeded adversarial fixture, and fail against the seeded adversarial case if the owning implementation is tampered to allow it." Representative bars: `sandbox-escape` (0 escapes across hostile capability/fuel/memory/host-tamper modules), `mesh-handoff` (200 entities × 10,000 crossings, zero loss/dup/reconnect, with a mid-handoff node death injected), `single-writer` (only the authoritative owner advances the ledger), `netcode-loss` (playable @5%, graceful @20%), `anticheat` (precision/recall vs a 95% bar on authoritative signals), `csam-hash` (100% synthetic known-hash match before serve), `payout-formula` (correct to the cent), `trust-boundary`/`identity-leak` (no platform secret or identity reachable from realm code or web view), and the nine gates added 2026-06-12 (`webview-escape`, `event-tamper`, `auth-tamper`, `capability-deny`, `ban-evasion`, `overload`, `character-deletion-limit`, `generation-policy`, `accessibility`). `.github/workflows/v7-ci.yml` runs a `verify-inventory` job, then 23 Rust gate steps plus the Mawu Studio safety gates (generation-policy, accessibility) and the Abundantia economy gates (payout-formula, payout-eligibility, fraud-graph, currency-firewall). ### Golden replays: the determinism guard Determinism is the cross-cutting invariant (rollback prediction, replay-based crash recovery, replayable plugins), so the golden-replay gate is its CI guard. `run_moremi_golden_replay_ci_gate` (`moremi-realm-server/src/lib.rs:9307`) runs two halves: the **golden gate** replays a recorded input log tick-by-tick through `moremi_state_hash` and asserts the recorded inputs reproduce the exact authoritative state hash, and the **nondeterminism probe** clones the fixture, flips the seed (`config.seed ^= 0xDEAD_BEEF`), and requires the hash to _diverge_ — so the gate fails if unrecorded entropy ever leaks in. The test `golden_replay_ci_gate_fails_unrecorded_entropy_probe` pins that property; the same capture machinery feeds incident-replay clips and photo-mode exports with content hashes signed by the authoritative node. ```mermaid flowchart TB fixtures["29 seeded adversarial fixtures
(hostile module · dying node · forged event · bot ring)"] fixtures --> evals["Rust + TS eval fns
run_*_eval / *_gate (apps/v7 · libs/v7)"] evals --> ci["v7-ci.yml
verify-inventory → 23 rust gates + studio + economy"] inv["verify-adversarial-eval-gates.mjs
inventory wiring"] --> ci ci --> review{"verify-launch-readiness.mjs
no-fabrication invariant"} rollout["verify-staged-rollout.mjs
advance iff met"] --> review review -->|"every signoff fixture-passed"| red["launch-readiness-review.json
decision: not-launch-ready"] review -.->|"only on real production signoff"| green["green = GA"] ``` Three verifiers back the gate machinery: `verify-adversarial-eval-gates.mjs` (the 29-gate inventory wiring), `verify-staged-rollout.mjs`, and `verify-launch-readiness.mjs`, which additionally **greps the source for the eval functions and their key assertions** (e.g. `assert_eq!(report.escaped_fixtures, 0);`, `assert_eq!(report.client_reconnects, 0);`) so a hand-edited evidence file cannot pass while the code that would produce it is absent. ## Launch readiness V7's go/no-go aggregates everything above, and its defining property is that it is **honestly not ready and structurally cannot lie about it.** `V7/release/launch-readiness-review.json` reads `decision: "not-launch-ready"`. All eight signoffs — trust boundary, Ixchel sandbox, Danu scale/handoff, Sekhmet child-safety/IR, creator economy, anti-cheat, moderation/minor-protection, the all-realms safety floor — carry `status: "fixture-passed"` and the explicit `evidenceBasis: "in-repo adversarial fixture; not a production measurement"`. The checked-in `correctionNote` is unsparing: the original 2026-06-02 "launch-ready" decision predated any adversarial audit (it was recorded three days after the feature spec was written); the 100%/100% precision-recall figures are "fixture-toy signatures" (6 true positives, 0 FP, 0 FN on seeded cases), not classifier performance against the 95% bar; the UE client is a ~2.2k-LOC skeleton; and PhotoDNA, PDQ, CSAI Match, Thorn Safer, Lantern, StopNCII, and EAC are **not integrated**. The staged-rollout record corroborates: all five stages (creator-alpha, rp-beta, meshing-beta, economy-launch, GA) are `status: "planned"` / `decision: "pending"`, and the incident-response drills recorded in them were **code-level simulations, not operational drills**. The machinery enforces this honesty. `verify-launch-readiness.mjs` encodes a **no-fabrication invariant**: if any signoff is `fixture-passed`, the decision _cannot_ be `launch-ready` (`:378`–`388`); the rollout verifier enforces **advance-iff-met** so a stage cannot claim it advanced without being marked met; and the review's own `verify:v7 launch-readiness` script, which once asserted the fabricated "launch-ready" record, now fails against it intentionally and must be reworked to verify the honest state before it is re-enabled as a gate. V7 will be launch-ready — per the architecture's checklist — only when the trust boundary is enforced against a real realm process and web view, the sandbox has zero escapes on a launch-scale corpus, Danu sustains 1000+ with clean handoffs on a real cluster, Sekhmet meets its CSAM/grooming/malware bars with the real vendor integrations and passes an operational IR drill, age-assurance and the ≤48 h NCII pipeline are live, payouts settle through Aje with the anti-fraud gates, and the Lilith/Kuanyin safety floor is proven enforced on every realm regardless of operator configuration. Until each of those flips on real evidence, the same verifier keeps the aggregate negative — which, pre-GA, is worth more than a green that lies. ## Where this connects - **Sideways:** [Hosting, Fleet, Data & Compliance](./hosting-fleet-data-and-compliance.md) owns the Agones realm fleet, residency tagging, and DSAR pipeline whose per-realm health this page's `/health` contract and cost-lever telemetry feed; [Thesis, Trust Boundary & Topology](./thesis-trust-boundary-and-topology.md) owns the platform/realm boundary the `trust-boundary`, `identity-leak`, and `webview-escape` gates defend and the safety floor the launch review signs off. - **Platform foundations it composes:** the [shared platform](../../platform/overview.html) observability libraries (`@oshun/metrics`, `@oshun/tracing` — real and platform-wide, inherited but not yet wired into the V7 Rust services), the V1 observability stack the gateway reuses, and the V5/V6 build pipeline V7's cook and patch path inherits. - The section hub: [../V7_ARCHITECTURE.md](../V7_ARCHITECTURE.md).