# Lilith Operations, Performance, Testing, and Release Lilith spans web/mobile/desktop services, Rust realtime workloads, GPU Pixel Streaming capacity, a cross-language protocol, and Unreal client targets. It therefore releases through evidence by surface and tier; one green unit suite or one modeled load report cannot certify the whole product. ```mermaid flowchart TD Source[Versioned source protocol content and configuration] --> Static[Lint type contract schema and migration gates] Static --> Unit[Unit property and cross-language protocol tests] Unit --> Integrated[World gateway voice identity tenant and provider integration] Integrated --> Clients[UE Pixel Streaming Tier 2 web mobile VR and accessibility journeys] Clients --> Scale[Load soak latency frame network GPU and regional capacity] Scale --> Trust[Security privacy moderation rights commerce and recovery] Trust --> Canary[Compatible deployment and bounded tenant cohort] Canary --> Observe{SLO quality safety and cost evidence agree?} Observe -- no --> Contain[Abort degrade roll back or forward-fix] Contain --> Integrated Observe -- yes --> Promote[Promote by tier region platform and tenant] Promote --> Maintain[Drift incident restore and content recall exercises] Maintain --> Source ``` Promotion is multi-dimensional: one tier, region, client, tenant, or service can remain gated while another is evidenced. The release record names exactly what passed. ## Operational topology | Layer | Key operational risks | | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Identity/BFF and shared V1 dependencies | Auth/tenant/release mismatch, memory/grounding/policy/commerce dependency failure. | | World server | Shard imbalance, persistence lag, physics abuse, entity/interest pressure, reconnect consistency. | | Realtime gateway and voice | Transport negotiation, session/resume, TURN/SFU capacity, packet loss/jitter, voice abuse and moderation. | | Pixel Streaming | GPU worker availability, match/admission, signaller/TURN, first-frame latency, cost, regional capacity. | | Tier 2 web | WebGPU/WebGL compatibility, bundle/first-load, browser memory/FPS, feature fallback parity. | | Unreal/native | Platform builds, optional plugins/assets, cook/package size, frame pacing, input/VR/accessibility, store certification. | | Tenant services | Class/concert scheduling, content readiness, rights/provenance, ticket/royalty consistency. | | Media/CDN | Asset bundle versioning, streaming/caching, takedown propagation, regional availability. | The operational source library includes [Operator Runbooks and On-call](../../OPERATOR_RUNBOOKS_ON_CALL.md), [Grafana Dashboards](../../GRAFANA_DASHBOARDS.md), and the V3 runbooks under `V3/runbooks/`. ## Observability Correlate a participant journey across launch decision, BFF/session, client tier, gateway connection, world/shard/room, voice realm, tenant activity, commerce/rights, and leave/reconnect. Stable correlation identifiers are more useful than copying raw personal or voice content into telemetry. Monitor: - launch decisions/reasons and fallback frequency by region/device; - auth/entitlement/room-join success and time; - gateway handshake, transport distribution, RTT/loss/jitter, reconnect and resume outcomes; - world tick/frame, entity and visible-set counts, physics rejection, shard load, persistence/cache/stream lag; - voice join, SFU/TURN health, speaking/mute state failures, and abuse actions; - Pixel Streaming match, queue, GPU utilization, first frame, session start, disconnect, and per-session cost; - UE and Tier 2 client FPS/frame time, memory, crashes, asset/cook/bundle load; - Tara class and Saraswati concert start/completion/sync/fallback; - rights, consent, provenance, review, payment, royalty, and takedown workflow lag/failures; - accessibility/localization mode failures and store/platform-specific defects. Dashboards should link to actionable runbooks, current SLO/error budget, and the deployment/version dimensions needed for rollback. ## Capacity and performance budgets Performance is tier- and workload-specific. Do not mix: - venue capacity with wire bandwidth or voice-realm tiers; - client frame budgets with server tick budgets; - modeled/synthetic validation with measured deployed fleet behavior; - median latency with tail/reconnect/session-start reliability; - empty-room FPS with representative avatar, VFX, audio, UI, and network load. Relevant evidence surfaces include [Fleet Load Validation](../../FLEET_LOAD_VALIDATION.md), [Pixel Streaming RTT Validation](../../PIXEL_STREAMING_RTT_VALIDATION.md), [Pixel Streaming Session Start Success](../../PIXEL_STREAMING_SESSION_START_SUCCESS.md), [Web First Load Budget](../../WEB_FIRST_LOAD_BUDGET.md), [UE Desktop FPS Validation](../../UE_DESKTOP_FPS_VALIDATION.md), and platform- specific mobile/VR/console reports. Where a report is a deterministic simulation/model, label it as such and retain the production measurement gate. ## Test layers | Layer | What it proves | | ------------------------------------ | ------------------------------------------------------------------------------------------------ | | Schema/contract fixtures | Valid shapes, enums, compatibility, error cases, and registry completeness. | | Unit/property tests | Domain logic, policy, codecs, calculations, idempotency, and edge cases. | | Cross-language protocol goldens | TypeScript/Rust/Unreal encode/decode compatibility. | | Service integration | Auth/tenant/data/event/tool/provider behavior against real dependencies. | | Rust load/fault tests | World/gateway/relay behavior under concurrency, reconnect, backpressure, and failure. | | Unreal automation/Gauntlet | Modules, maps/modes, multiplayer, performance, packaging, and client behavior. | | Web Playwright and mobile automation | Launch, Tier 2, studio/admin/customer flows, accessibility, and device interaction. | | Media/rights/quality gates | Artifact technical quality, sync, identity, provenance, consent, similarity, and release policy. | | Certification and store gates | Platform requirements, packages, privacy/legal assets, and human/vendor approvals. | Tests should use representative room/content bundles and failure cases. A compile-only Unreal module or passing contract fixture is not end-to-end feature coverage. ## Critical end-to-end journeys - account/entitlement → launch decision → chosen client → room join → leave and reconnect; - Tara instructor/class creation → review/schedule → participant class → Aja cues → completion/continuity; - Saraswati persona/track → rights/provenance review → concert authoring → prewarm/live/fallback → royalties; - Lilith Studio scene/persona/asset publish with rejected, exception, conflict, rollback, and takedown paths; - recording/voice consent grant, visible capture, revocation, and derivative propagation; - harassment/voice abuse report → immediate protection → operator review → notice/appeal; - Pixel Streaming capacity/region failure → honest Tier 2/static fallback; - protocol/client-version mismatch → update or compatible fallback; - region/residency and DSAR workflows across V1 and V3 records. The repository walkthrough index includes the [Lilith Studio Tara scene publish journey](../../../WALKTHROUGH/journeys/lilith-studio-tara-scene-publish.md). ## Deployment and rollback Deploy contracts and backward-compatible services before clients that depend on them. Coordinate world/gateway/protocol and content bundle versions; drain realtime/GPU sessions rather than terminating blindly; migrate tenant/commerce/ rights data with reconciliation; canary by region/tier/tenant; and preserve a known compatible client/service/content matrix. Rollback must account for irreversible or externally visible actions such as payments, royalties, published media, provenance assertions, store releases, and takedowns. These use compensating/reconciliation workflows rather than a simple binary rollback. ## Incident response 1. Establish affected region, tenant, room/event, client tier/version, service, content bundle, and policy/release. 2. Protect participants first: block joins, disable capture/voice/tool, freeze release/commerce, or route to a safe fallback as narrowly as possible. 3. Preserve correlated logs/metrics/traces and authoritative receipts with privacy/redaction controls. 4. Check cross-tenant, rights, consent, minor-safety, and payment blast radius. 5. Reproduce with the smallest relevant contract/integration/load/client test. 6. Recover from authoritative state; reconcile events, payments, provenance, takedowns, and agent/operator actions. 7. Add the failure to automation and update the runbook/gate before closure. The release-grade security procedure is [Security Incident Response](../../runbooks/security-incident-response.md). ## Release gate A release candidate identifies and passes the applicable: - contract/protocol/schema and migration gates; - service, Rust, web, mobile, and Unreal build/test targets; - tier-specific performance, reconnect, load, and packaging budgets; - accessibility and localization automation plus required human review; - persona/safety, consent, rights, provenance, moderation, and data-protection gates; - studio/tenant/customer/admin critical journeys; - regional/provider/GPU capacity and observability readiness; - store/platform/legal/vendor/human gates; - rollback, incident, backup/restore, and on-call readiness. Outstanding exceptions are scoped, owned, time-bounded, and visible in the go/no-go record. A planned inventory target, vendor dependency, or unavailable production fleet cannot be reworded into a pass. Start with [Launch Readiness](../../LAUNCH_READINESS.md), [Testing, QA, and Accessibility](../../TESTING_QA_ACCESSIBILITY.md), and [Cross-platform Certification](../../CROSS_PLATFORM_CERTIFICATION.md). ## Related - [Metaverse worlds and presence](./metaverse-worlds-and-presence.md) - [Platform services, data, and integrations](./platform-services-data-and-integrations.md) - [Operations, rollout, and launch](../../features/operations-rollout-and-launch.md) - [Observability, performance, security, and launch architecture](../../architecture/observability-performance-security-and-launch.md)