Lilith Metaverse · Architecture

V3 Product Promise, Tiered Client Stack, and High-Level Architecture

A focused page within the Lilith Metaverse Architecture documentation. The full map and every sibling page live in the Architecture hub.

9sections12 minread1diagram
← Back to the Architecture hubMarkdown source
On this page

V3 — "Lilith" — is Oshun's embodied, multi-user surface: one 3D space that carries V1's identity, memory, grounding, safety, provenance, and rights guarantees into a real-time world where people practice yoga together (Tara Studio), attend concerts by persistent AI artists (Saraswati Stage), and gather in shared civic rooms (Lilith Commons). The defining architectural choice is a tiered client stack with Unreal Engine 5.5 as the canonical runtime and a reduced-fidelity browser client as the floor, so that no device is locked out at the door. That choice is not aspirational prose — it is wired through the repository: the UE project at V3/ue/V3.uproject pins "EngineAssociation": "5.5" and "DisableEnginePluginsByDefault": true, declares 17 C++ modules and a dedicated V3PixelStreamingWorker.Target.cs server-render target, and enables PixelStreaming, GameplayAbilities, Niagara, Metasound, Mover/PoseSearch, the Iris replication system, OpenXR, and EOS; the Tier-2 client is a real Next.js app at apps/v3/lilith-web-fallback/ rendering through FallbackThreeCanvas.tsx; and three authoritative Rust services (apps/v3/lilith-world-server, lilith-realtime-gateway, lilith-pxstream-relay) share a single cross-language wire protocol. This page is the orientation hub for how those pieces fit; the section index is ../V3_ARCHITECTURE.md.

The reason the stack is built this way is economic and physical, not stylistic. The premium experience — Lumen GI, Nanite geometry, MetaHumans, Sequencer-driven concerts — can only run inside UE5, so a single UE project drives every premium surface (native desktop/VR/console/mobile and the browser via server-rendered Pixel Streaming over WebRTC). But server-rendered video needs a GPU worker per session and a nearby POP, which is not always available; so a second, locally-rendered three.js/WebGPU client exists for constrained devices, constrained bandwidth, and regions without Pixel Streaming capacity. Both clients speak the same multiplayer wire protocol to the same world server and reuse the same V1 BFF — the only thing that degrades across tiers is rendered fidelity, never identity, presence, or safety. The launch decision that routes a visitor to one of those surfaces is itself a typed contract (LilithLaunchDecisionSchema in libs/contracts/src/v3/lilith.ts), with four possible surfaces and twenty-one reason codes (a single decision carries up to sixteen).

What ships, honestly#

The wire protocol, the contract model, the three Rust services, and the UE C++ module skeleton are real and tested. One Protobuf schema (libs/v3/multiplayer-protocol/proto/oshun/v3/multiplayer/v1/multiplayer.proto) is implemented three times — a ts-proto/TypeScript codec, a prost/Rust codec, and a hand-written UE C++ varint/zig-zag codec in V3/ue/Source/V3Net/Private/V3NetProtocol.cpp — and a cross-language golden-hex test (SNAPSHOT_DELTA_GOLDEN_HEX in libs/v3/multiplayer-protocol/rust/src/lib.rs) proves the three encoders agree byte-for-byte on the canonical fixture. The world server, gateway, and Pixel Streaming relay are substantial Rust (consistent-hash sharding, an rstar R-tree interest manager, rapier3d physics validation, JWT refresh-token rotation with reuse detection, a KEDA/Karpenter-emitting autoscaler) under a unsafe_code = "forbid" workspace, each with its own main.rs and an axum /healthz. libs/contracts/src/v3 ships 30 registered Zod contracts (V3_CONTRACT_REGISTRY in registry.ts) with validated fixtures.

Some surfaces are modeled, not live. The fleet-load and reconnect "validation" reports in the Rust services compute deterministic phase-timing models (base latency plus a seeded jitter term — see simulate_transient_reconnect_validation in the gateway), even though the underlying reconnect/resume and shard-pin logic they exercise is real. The Pixel Streaming relay is a genuine matchmaker, admission gate, and Epic-signaller relay — build_epic_signaller_answer_sdp synthesizes a valid WebRTC answer SDP with HMAC-derived ICE credentials — but the actual UE-rendered video frames require a deployed GPU worker fleet that this repository plans for but does not run. Finally, the heavy creative payload is vendor/art-gated: the V3.uproject lists MetaHumanRuntime, MetaHumanSDK, LiveLinkFaceImporter, VRM4U, ResonanceAudio, and SteamAudio as "Optional": true plugins, Wwise as disabled, and the 129 tracked .uasset binaries are overwhelmingly generated design-token UI assets (64 Styles, 40 Brushes, 3 Fonts under Content/UI/DesignTokens/Generated/) plus the 21 GameFeatureData.uasset plugin descriptors — there are no hand-authored MetaHuman, venue, or concert art assets in-tree. Where a claim depends on those, this page says so.

The product promise, concretely#

Three tenants ride one substrate. The fixed vocabulary is enumerated in V3TenantSchema (libs/contracts/src/v3/primitives.ts) as exactly four tenant ids — lilith-platform, lilith-commons, tara-studio, saraswati-stage — and each maps to a TypeScript adapter package (@oshun/tenant-tara-studio, @oshun/tenant-saraswati-stage, @oshun/tenant-lilith-commons) and a set of UE Game Feature plugins. The promise is that the substrate, not each tenant, provides embodiment: a V1 Oshun account binds to an avatar (AvatarBindingSchema, with a swapCooldownUntil enforcing the 24-hour swap cooldown defined as AVATAR_SWAP_COOLDOWN_MS in the world server), presence is authoritative and visibility-banded, and rights/provenance travel with every captured artifact (ProvenanceBundle3DSchema extends the V1 provenance bundle with performanceId, venueId, and a 64-hex musicSyncDigest).

The tiered client stack#

Tier 1 — the canonical UE5 client (V3/ue/)#

A single UE5.5 project targets every premium surface. Its 17 modules (V3.uproject) split responsibilities cleanly: V3Core (subsystems, save-game, the V1-account bridge), V3World (room replication, scene streaming, interest management — it depends on Json and V3UI per V3/ue/Source/V3World/V3World.Build.cs), V3Net (the wire-format codec), V3Avatar/V3Animation (VRM + MetaHuman, motion matching via the Mover and PoseSearch plugins), V3Voice/V3Audio (SFU client + MetaSounds), V3VFX (Niagara), V3Cinematics (Sequencer), V3OnlineServices (EOS), plus V3Editor (editor-only) and V3Tests (a DeveloperTool module for Gauntlet automation). Across V3/ue/Source there are 73 .cpp and 40 .h files. The project enables the full premium plugin set — PixelStreaming(+Player), CommonUI + ModelViewViewModel, GameplayAbilities, MotionWarping/AnimationWarping, ChaosClothAsset, ReplicationGraph + NetworkPrediction + Iris, OpenXR(+HandTracking), Gauntlet, and MovieRenderPipeline — and gates the art/audio middleware behind "Optional": true. Crucially, V3PixelStreamingWorker.Target.cs is a separate build target: the same project compiles both the player-facing client and the headless server-render worker that streams frames to browsers.

Modularity is delivered through 21 Game Feature plugins under V3/ue/Plugins/GameFeatures/ — 3 tenant plugins (V3Tenant_TaraStudio, V3Tenant_SaraswatiStage, V3Tenant_LilithCommons) and 18 mode plugins (V3Mode_TaraLiveClass, V3Mode_SaraswatiConcert, V3Mode_CommonsObservatory, …). Each ships a real GameFeatureData.uasset so it can be hot-activated per room. See ./subsystem-glossary-and-layout.md for the full module/plugin map and ./tier1-ue5-client.md for the client architecture.

Tier 2 — the lightweight web fallback (apps/v3/lilith-web-fallback/)#

The fallback is a real Next.js app, not a placeholder. Its root src/app/page.tsx is a one-line redirect('/commons'); the actual experience lives at src/app/[tenant]/page.tsx, which renders FallbackExperience.tsx over a WebGPU/WebGL2 canvas (FallbackThreeCanvas.tsx). It renders locally at reduced fidelity (selectable webgpu/webgl2 renderer with per-renderer FPS budgets from lilithFallbackRendererBudgets), connects to the same gateway via connectLilithGatewayWithFallback (degrading WebTransport → WebRTC → WebSocket by network profile), runs a rapier-based prop-pickup prediction check, and drives spatial audio through @oshun/spatial-audio (HRTF when available, flat stereo when toggled off). Tenant features degrade gracefully rather than vanish: Saraswati stadium concerts surface as a degraded preview tier (LilithSaraswatiFallbackTier of 'hall' etc.) with a Yemaya proxy URL, while Tara classes and Commons rooms keep full interactivity. The same component also composes the launch-accessibility reports (reduced motion, color-vision palettes, photosensitive-safe mode, single-switch navigation, launch localization) from @oshun/tenant-lilith-commons. See ./tier2-fallback-web-client.md.

The browser-via-Pixel-Streaming entry (apps/v3/lilith-web/)#

Between the two tiers sits the thin browser shell that prefers Tier 1 by streaming server-rendered UE frames. LilithLanding.tsx offers three launch modes (Pixel Streaming, native deep-link, fallback WebGL), and on "Enter Lilith" it POSTs to /api/v3/pxstream/match and drives an @oshun/lilith-web-pxstream client over createEpicPixelStreamingAdapter(), reporting first-frame latency, mic permission, and reconnect state. This is the "web via Pixel Streaming" surface — the browser sees full UE fidelity because a GPU worker is rendering it remotely. Tier selection across native/pxstream/ fallback/static is the dedicated subject of ./tier-routing-and-pixel-streaming.md.

One honest wrinkle: "tiers" mean different things at different layers#

The word tier is overloaded across the codebase, and the docs-center should be precise about it. The contract V3CapacityTierSchema defines four venue sizes — class, salon, theater, stadium. The wire protocol CapacityTier enum collapses to two — CLASS and STADIUM — and the only per-tier bandwidth budgets that exist are CLASS_TIER_MAX_BPS = 32_000 and STADIUM_TIER_MAX_BPS = 256_000 (multiplayer-protocol crate). The gateway introduces a third taxonomy, VoiceRealmTier { Class, Hall, Stadium }, for voice policy. And the world server's interest manager caps visible entities at CLASS_TIER_VISIBLE_ENTITY_CAP = 32 vs STADIUM_TIER_VISIBLE_ENTITY_CAP = 256. None of these contradict each other, but they are not the same axis: contract tiers describe a venue's authored size; wire/interest tiers describe the bandwidth-and-LOD regime; voice tiers describe crowd-audio policy. Treat "tier" as layer-scoped.

The shared wire protocol — the spine that makes the stack coherent#

Everything above only works because all clients speak one protocol. The schema (multiplayer.proto, package oshun.v3.multiplayer.v1) quantizes everything to integers for compactness and determinism: positions are Vector3Mm (signed millimeters, zig-zag varint), rotations are RotationMilliDegrees, expression intensity is basis points. It defines snapshot and delta packets, presence, voice control, interaction, gameplay-action, and operator-control messages, plus ClientEnvelope/ServerEnvelope with oneof payloads and a VersionNegotiation handshake.

The protocol is implemented in all three runtimes from that one schema:

  • TypeScript (libs/v3/multiplayer-protocol/src/index.ts) re-exports the generated codec and adds applySnapshotDelta, predictSnapshot (dead-reckoning clamped to 250 ms), negotiateProtocolVersion, and a runDecoderBenchmark.
  • Rust (.../rust/src/lib.rs) uses prost (wire module) plus encode_snapshot_delta/apply_snapshot_delta, validate_bandwidth_budget, and negotiate_protocol_version.
  • UE C++ (V3NetProtocol.cpp) hand-rolls the varint/zig-zag wire encoding — FV3NetMultiplayerProtocolCodec::EncodeSnapshotDeltaPacket and the matching decoders — and BuildInEngineProtocolExchangeReport() round-trips the full packet set in-engine.

The interop guarantee is a golden hex string asserted in the Rust tests: protobuf_snapshot_delta_matches_cross_language_golden_wire checks the canonical delta (room lilith-commons-atrium, base sequence 6, sequence 7, a calm-smile expression at 7,300 bp) encodes to a fixed 131-byte hex blob — the same fixture the TS canonicalSnapshotDeltaFixture() and the C++ CanonicalSnapshotDeltaFixture() build. Protocol negotiation selects the highest mutually-supported version and otherwise returns unsupported_version. See ./netcode-protocol-and-physics.md.

The authoritative service plane (Rust)#

Three services form a Cargo workspace (apps/v3/Cargo.toml, resolver 2, Rust 1.82, unsafe_code = "forbid"), each with a SERVICE_DESCRIPTOR, capability list, and main.rs.

Lilith World Server (port 43101, lilith-world-server/src/lib.rs) is the authority. It holds rooms in a RoomRegistry, routes them to shards with a ConsistentHashRing (virtual nodes + stable hashing) and gossips cross-shard chat over a CrossShardEventBus. Interest management is a real spatial query — an rstar R-tree over participant positions, returning the nearest N entities capped per tier. Movement is validated against a TELEPORT_SPEED_LIMIT_MPS = 12.0 ceiling, and interactive objects use server-validated pickup/place (InteractionAuthority). Presence visibility is a five-band model (VisibilityBand::{Public, Tenant, Cohort, Invited, Invisible}) computed against viewer profiles. The fixed-tick loop (TickLoopConfig) defaults to 50 Hz simulation, 20 Hz transform broadcast, 60 Hz expression interpolation, under a 5 ms p99 budget, and durable state splits across a Postgres durable store and a Redis hot-state stream (durable_persistence module). Packet processing emits OTel/Jaeger spans (process_join_packet_with_tracing).

Lilith Realtime Gateway (port 43102, lilith-realtime-gateway/src/lib.rs) is the edge. It terminates three transports with graceful fallback (WebClientNetworkProfile: UDP → TURN-TCP → WebSocket), authenticates handshakes with HS256 JWTs, and runs a full refresh-token rotation store with reuse detection and family revocation (rotate_refresh_token revokes the whole token family on replay; 15-minute access, 30-day refresh). It pins sessions to shards (SessionPinDirectory, 60 s TTL) and resumes transient drops while preserving shard pin and server sequence (GatewayReconnectDirectory). Its Voice SFU is LiveKit-compatible, Opus 24 kbps mono / 20 ms frames, with HRTF listener positioning and operator mute/kick; per-tenant voice policy is concrete — VoiceChatPolicyDirectory::lilith_defaults() gives Tara direct voice with instructor mute at every tier, and Saraswati disabled at class/hall but a crowd-bed proxy at stadium.

Lilith Pixel Streaming Relay (port 43103, lilith-pxstream-relay/src/lib.rs) is the browser-fidelity broker. It matches a request to a POP by RTT, utilization, codec (H.264/AV1), and residency; issues a short-lived session JWT (HMAC, from V3_PXSTREAM_JWT_HMAC_SECRET); brokers the Epic signaller SDP exchange; and plans GPU capacity with a plan_pxstream_worker_scale that emits real KEDA ScaledObject and Karpenter NodePool YAML, plus scheduled-concert prewarm (15-minute asset lead, 30-minute worker lead) and multi-cloud pools (AWS/Azure/GCP/CoreWeave/Lambda). Its admission gate is a domain model, not a gate-stub: a PxStreamAdmissionDecision with seven actions (allow/deny/fallback/ prompt-idle/disconnect-idle/refer-safety/ban) backed by an input-entropy bot classifier (precision floor 90%), free-tier minutes (120/day), and per-user (2), per-network (8), and institutional (64) session caps. See ./world-server-and-gateway.md.

Data, contracts, and tenancy#

The V3 data plane is contract-first. libs/contracts/src/v3 defines primitives (V3UuidSchema, V3Vector3Schema, V3RoyaltySplitSchema that must sum to 10,000 bps, V3CitationRefSchema) and four domain files (lilith.ts, tara.ts, saraswati.ts, commons.ts) feeding a single V3_CONTRACT_REGISTRY of 30 contracts — 8 lilith-platform, 8 tara, 8 saraswati, 5 commons, 1 cross-cutting EmbodiedConsent. Each descriptor carries a Zod schema, a validated fixture, a route segment, and a Prisma model name (V3<ContractName>) pointing at one of three tenant Prisma schemas. Tenancy is enforced both in data (databases are domain-isolated) and at the body — presence is filtered by visibility band in the world server before broadcast, so a Commons "solitary cell" participant is structurally invisible, not merely hidden in UI. Residency and DSAR ride the V1 substrate. See ./data-tenancy-and-residency.md and ./tara-classes-aja-and-commons.md.

A worked example: cold-open to first frame#

The launch decision is modeled as a contract (LilithLaunchDecisionSchema), and the four-surface routing (native, pxstream, fallback, static) plays out across the BFF, the relay, and the world server:

sequenceDiagram participant U as Browser participant BFF as V1 BFF participant R as PxStream Relay (43103) participant W as UE Worker (server-render) participant G as Realtime Gateway (43102) participant WS as World Server (43101) U->>BFF: GET catalog / identity / entitlement (HTTPS) Note over U: probe webRtc / codecs / webGpu / downlink U->>R: POST /api/v3/pxstream/match (region, codec, residency) R->>R: admission gate + POP scoring + session JWT R-->>U: match decision (POP, signaller WS, first-frame budget) U->>R: POST /signalling/exchange (SDP offer + ICE) R-->>U: synthesized answer SDP (HMAC-derived ICE creds) U-->>W: WebRTC media (server-rendered UE frames) U->>G: handshake (Bearer access JWT, room id) G->>G: verify JWT, pin session to shard (60s TTL) G->>WS: join_room(participant) WS-->>G: snapshot + visibility-filtered presence G-->>U: snapshot delta stream (20 Hz), voice SFU (Opus)

If the browser lacks WebRTC or no POP is within the latency budget, the same decision contract routes to the fallback-browser target (WebGPU/WebGL2, reduced fidelity); if even local rendering is unsupported, it routes to static-landing with reason local-rendering-unsupported. Native clients skip the relay and go straight to the gateway. Failure modes are first-class: the relay returns typed errors (PXSTREAM_NO_POP_AVAILABLE → 503, PXSTREAM_SESSION_JWT_INVALID → 401), the gateway rejects replayed refresh tokens by revoking the token family, and the world server rejects teleport-speed jumps and out-of-band interaction requests.

How it connects to neighbouring systems#

V3 layers on V1 without forking it: clients hit the V1 BFF over HTTPS for catalog, identity, billing, and scheduling, while the world plane handles real-time state. Cross-domain extensions reuse V1 packages — @oshun/aja-pose for in-world pose coaching, @oshun/sophia-saraswati-grounding for artist backstory citations, Aje for tickets and royalties, Iris for spatial memory scopes. The neighbouring docs-center pages cover each seam in depth: ./avatar-animation-and-audio.md, ./saraswati-stage-pipeline.md, ./authoring-and-content-pipeline.md, ./v1-integration-and-identity-bridge.md, ./persona-policy-provenance-and-rights.md, ./commerce-and-royalties.md, and ./observability-performance-security-and-launch.md.