Lilith spans web/mobile/desktop services, Rust realtime workloads, GPU Pixel Streaming capacity, a cross-language protocol, and Unreal client targets. It therefore releases through evidence by surface and tier; one green unit suite or one modeled load report cannot certify the whole product.
Promotion is multi-dimensional: one tier, region, client, tenant, or service can remain gated while another is evidenced. The release record names exactly what passed.
Operational topology#
| Layer | Key operational risks |
|---|---|
| Identity/BFF and shared V1 dependencies | Auth/tenant/release mismatch, memory/grounding/policy/commerce dependency failure. |
| World server | Shard imbalance, persistence lag, physics abuse, entity/interest pressure, reconnect consistency. |
| Realtime gateway and voice | Transport negotiation, session/resume, TURN/SFU capacity, packet loss/jitter, voice abuse and moderation. |
| Pixel Streaming | GPU worker availability, match/admission, signaller/TURN, first-frame latency, cost, regional capacity. |
| Tier 2 web | WebGPU/WebGL compatibility, bundle/first-load, browser memory/FPS, feature fallback parity. |
| Unreal/native | Platform builds, optional plugins/assets, cook/package size, frame pacing, input/VR/accessibility, store certification. |
| Tenant services | Class/concert scheduling, content readiness, rights/provenance, ticket/royalty consistency. |
| Media/CDN | Asset bundle versioning, streaming/caching, takedown propagation, regional availability. |
The operational source library includes
Operator Runbooks and On-call,
Grafana Dashboards, and the V3 runbooks under
V3/runbooks/.
Observability#
Correlate a participant journey across launch decision, BFF/session, client tier, gateway connection, world/shard/room, voice realm, tenant activity, commerce/rights, and leave/reconnect. Stable correlation identifiers are more useful than copying raw personal or voice content into telemetry.
Monitor:
- launch decisions/reasons and fallback frequency by region/device;
- auth/entitlement/room-join success and time;
- gateway handshake, transport distribution, RTT/loss/jitter, reconnect and resume outcomes;
- world tick/frame, entity and visible-set counts, physics rejection, shard load, persistence/cache/stream lag;
- voice join, SFU/TURN health, speaking/mute state failures, and abuse actions;
- Pixel Streaming match, queue, GPU utilization, first frame, session start, disconnect, and per-session cost;
- UE and Tier 2 client FPS/frame time, memory, crashes, asset/cook/bundle load;
- Tara class and Saraswati concert start/completion/sync/fallback;
- rights, consent, provenance, review, payment, royalty, and takedown workflow lag/failures;
- accessibility/localization mode failures and store/platform-specific defects.
Dashboards should link to actionable runbooks, current SLO/error budget, and the deployment/version dimensions needed for rollback.
Capacity and performance budgets#
Performance is tier- and workload-specific. Do not mix:
- venue capacity with wire bandwidth or voice-realm tiers;
- client frame budgets with server tick budgets;
- modeled/synthetic validation with measured deployed fleet behavior;
- median latency with tail/reconnect/session-start reliability;
- empty-room FPS with representative avatar, VFX, audio, UI, and network load.
Relevant evidence surfaces include Fleet Load Validation, Pixel Streaming RTT Validation, Pixel Streaming Session Start Success, Web First Load Budget, UE Desktop FPS Validation, and platform- specific mobile/VR/console reports.
Where a report is a deterministic simulation/model, label it as such and retain the production measurement gate.
Test layers#
| Layer | What it proves |
|---|---|
| Schema/contract fixtures | Valid shapes, enums, compatibility, error cases, and registry completeness. |
| Unit/property tests | Domain logic, policy, codecs, calculations, idempotency, and edge cases. |
| Cross-language protocol goldens | TypeScript/Rust/Unreal encode/decode compatibility. |
| Service integration | Auth/tenant/data/event/tool/provider behavior against real dependencies. |
| Rust load/fault tests | World/gateway/relay behavior under concurrency, reconnect, backpressure, and failure. |
| Unreal automation/Gauntlet | Modules, maps/modes, multiplayer, performance, packaging, and client behavior. |
| Web Playwright and mobile automation | Launch, Tier 2, studio/admin/customer flows, accessibility, and device interaction. |
| Media/rights/quality gates | Artifact technical quality, sync, identity, provenance, consent, similarity, and release policy. |
| Certification and store gates | Platform requirements, packages, privacy/legal assets, and human/vendor approvals. |
Tests should use representative room/content bundles and failure cases. A compile-only Unreal module or passing contract fixture is not end-to-end feature coverage.
Critical end-to-end journeys#
- account/entitlement → launch decision → chosen client → room join → leave and reconnect;
- Tara instructor/class creation → review/schedule → participant class → Aja cues → completion/continuity;
- Saraswati persona/track → rights/provenance review → concert authoring → prewarm/live/fallback → royalties;
- Lilith Studio scene/persona/asset publish with rejected, exception, conflict, rollback, and takedown paths;
- recording/voice consent grant, visible capture, revocation, and derivative propagation;
- harassment/voice abuse report → immediate protection → operator review → notice/appeal;
- Pixel Streaming capacity/region failure → honest Tier 2/static fallback;
- protocol/client-version mismatch → update or compatible fallback;
- region/residency and DSAR workflows across V1 and V3 records.
The repository walkthrough index includes the Lilith Studio Tara scene publish journey.
Deployment and rollback#
Deploy contracts and backward-compatible services before clients that depend on them. Coordinate world/gateway/protocol and content bundle versions; drain realtime/GPU sessions rather than terminating blindly; migrate tenant/commerce/ rights data with reconciliation; canary by region/tier/tenant; and preserve a known compatible client/service/content matrix.
Rollback must account for irreversible or externally visible actions such as payments, royalties, published media, provenance assertions, store releases, and takedowns. These use compensating/reconciliation workflows rather than a simple binary rollback.
Incident response#
- Establish affected region, tenant, room/event, client tier/version, service, content bundle, and policy/release.
- Protect participants first: block joins, disable capture/voice/tool, freeze release/commerce, or route to a safe fallback as narrowly as possible.
- Preserve correlated logs/metrics/traces and authoritative receipts with privacy/redaction controls.
- Check cross-tenant, rights, consent, minor-safety, and payment blast radius.
- Reproduce with the smallest relevant contract/integration/load/client test.
- Recover from authoritative state; reconcile events, payments, provenance, takedowns, and agent/operator actions.
- Add the failure to automation and update the runbook/gate before closure.
The release-grade security procedure is Security Incident Response.
Release gate#
A release candidate identifies and passes the applicable:
- contract/protocol/schema and migration gates;
- service, Rust, web, mobile, and Unreal build/test targets;
- tier-specific performance, reconnect, load, and packaging budgets;
- accessibility and localization automation plus required human review;
- persona/safety, consent, rights, provenance, moderation, and data-protection gates;
- studio/tenant/customer/admin critical journeys;
- regional/provider/GPU capacity and observability readiness;
- store/platform/legal/vendor/human gates;
- rollback, incident, backup/restore, and on-call readiness.
Outstanding exceptions are scoped, owned, time-bounded, and visible in the go/no-go record. A planned inventory target, vendor dependency, or unavailable production fleet cannot be reworded into a pass.
Start with Launch Readiness, Testing, QA, and Accessibility, and Cross-platform Certification.