Lilith Metaverse · Runbooks

V3 GA Incident Response Runbooks

85% for 5 minutes.

6sections4 minread
Markdown source
On this page

runbook-set: v3-ga-incident-response.v1

verifier: pnpm verify:v3 operator-runbooks-on-call

Evidence caveat (added 2026-06-12). The six drill records below are self-attested: each carries only a date and a bare evidence ID, with no transcript, timing log, or participant roster behind it. Until each drill is rerun and captured with real artifacts (run log with timestamps, named participants, observed time-to-acknowledge / time-to-mitigate against the SLAs), treat these as tabletop/structural drills — the runbook content is real, but "Drill: green" is not operational proof. Incident-commander identifiers of the ic-na-01 style used elsewhere in the operations evidence denote on-call roles, not staffed, named humans.

Every launch incident below has a documented procedure, a named incident commander role, and writes an audit event into the Lilith Operator Console.

incident: room-overload#

  • Owner: Lilith-Safety incident commander
  • Severity: SEV-2, SEV-1 if paid event admission is blocked
  • Trigger: world-server room admission p95 exceeds 2 s or room shard CPU exceeds 85% for 5 minutes.
  • First action SLA: acknowledge within 2 minutes; cap new walk-ups within 5 minutes.
  • Steps: freeze room mutations, enable queue banner, split overflow into fallback web, assign operator host, and publish incident note.
  • Rollback: reopen walk-ups only after room admission p95 stays below 1 s for 15 minutes.
  • Escalation: Capacity / Pixel Streaming on-call, then world-server lead.
  • Audit event: operator.incident.room_overload.drilled
  • Drill: green, executed 2026-05-20, evidence drill-room-overload-ga

incident: generation-provider-outage#

  • Owner: Isis generation incident commander
  • Severity: SEV-2, SEV-1 if scheduled concert publication is blocked
  • Trigger: primary music, motion, or image provider error rate exceeds 10% for 5 minutes or queue p95 age exceeds 30 minutes.
  • First action SLA: acknowledge within 3 minutes; fail over provider lane within 10 minutes.
  • Steps: pause non-paid generation, switch approved jobs to backup provider, preserve provenance bundle, and notify editorial owners.
  • Rollback: restore primary provider only after 30 minutes of green canary jobs.
  • Escalation: Isis platform lead, Rights lead for provenance exceptions.
  • Audit event: operator.incident.generation_provider_outage.drilled
  • Drill: green, executed 2026-05-20, evidence drill-generation-provider-ga

incident: voice-sfu-degradation#

  • Owner: Spatial audio incident commander
  • Severity: SEV-2, SEV-1 if instructors or performers cannot be heard
  • Trigger: SFU packet loss exceeds 3%, jitter exceeds 60 ms, or voice join success drops below 99%.
  • First action SLA: acknowledge within 2 minutes; route affected rooms within 5 minutes.
  • Steps: move rooms to alternate SFU region, mute degraded uplinks, enable text fallback banner, and hand instructor host controls to operator.
  • Rollback: return rooms to primary SFU after 20 minutes of packet loss below 1%.
  • Escalation: Voice platform on-call, Capacity engineer.
  • Audit event: operator.incident.voice_sfu_degradation.drilled
  • Drill: green, executed 2026-05-21, evidence drill-voice-sfu-ga

incident: royalty-settlement-failure#

  • Owner: Lilith-Commerce incident commander
  • Severity: SEV-3, SEV-2 if artist payouts or refunds are delayed
  • Trigger: settlement job failure, payment ledger imbalance, or rights waterfall queue p95 exceeds 4 hours.
  • First action SLA: acknowledge within 10 minutes; freeze affected settlement batch within 15 minutes.
  • Steps: stop payout batch, reconcile signed receipts, notify Rights and Commerce pools, and publish artist-facing status note.
  • Rollback: release settlement only after dual-control ledger reconciliation.
  • Escalation: Finance operations, Rights lead, payment-provider TAM.
  • Audit event: operator.incident.royalty_settlement_failure.drilled
  • Drill: green, executed 2026-05-21, evidence drill-royalty-settlement-ga

incident: pixel-streaming-pop-outage#

  • Owner: Capacity / Pixel Streaming incident commander
  • Severity: SEV-2, SEV-1 during scheduled concert window
  • Trigger: POP session-start success below 99%, GPU quota exhaustion, or provider health check failure.
  • First action SLA: acknowledge within 2 minutes; fail over new sessions within 5 minutes.
  • Steps: drain affected POP, preserve paid-attendee priority, redirect free-tier traffic to fallback web, and open secondary-cloud capacity.
  • Rollback: shift traffic back only after 30 minutes of green synthetic session starts.
  • Escalation: Capacity engineering, cloud TAM, festival commander.
  • Audit event: operator.incident.pixel_streaming_pop_outage.drilled
  • Drill: green, executed 2026-05-21, evidence drill-pxstream-pop-ga

incident: master-capture-worker-failure-mid-concert#

  • Owner: Concert capture incident commander
  • Severity: SEV-1 for public concert, SEV-2 for rehearsal
  • Trigger: master capture worker heartbeat missing for 15 seconds, segment checksum mismatch, or C2PA signing failure.
  • First action SLA: acknowledge within 1 minute; promote hot standby worker within 3 minutes.
  • Steps: mark failed worker read-only, promote standby, replay last signed segment, verify consent and privacy ledgers, and notify editorial review gate.
  • Rollback: keep standby primary until post-show integrity review closes.
  • Escalation: Concert engineering lead, Rights lead, editorial producer.
  • Audit event: operator.incident.master_capture_worker_failure.drilled
  • Drill: green, executed 2026-05-22, evidence drill-master-capture-ga