Runbook - V6 Capacity Management

Use this runbook when the Moirai cluster, Pixel Streaming fleet, Tier-2 fallback, or Commons event load approaches capacity limits. The goal is to degrade gracefully before cognition deadlines or first-frame budgets fail.

Scope#

Triggers:

• Egbe Operator Console surface egbe-operator-surface-capacity-001 is degraded or critical.
• Audit action v6.egbe.capacity.acknowledge_pressure is required.
• Moirai load is above 85%, Clotho misses are above 2%, or Pixel Streaming sessions exceed the festival threshold.
• A Commons festival, Aye campaign, or district rollout increases active shards.
• verify:v6 commons-scale-load, verify:v6 cost-budget, or latency gates fail against current staging telemetry.

Applies to egbe-moirai-cluster, egbe-world-server, egbe-realtime-gateway, egbe-pxstream-relay, Tier-2 fallback web, Pixel Streaming workers, and the V1 audit platform.

Severity classification#

Detection signals#

• Audit action: v6.egbe.capacity.acknowledge_pressure.
• Capacity refs: moirai-cluster:orun-region-01:pressure-sample, pixel-streaming-fleet:orun-region-01:session-sample, and capacity-plan:commons-festival-tier:gathering.
• Governance refs: moirai-policy:cognition-capacity-pressure, pixel-streaming-policy:fleet-saturation-response, and v6-policy:graceful-degradation-required.
• Metrics: Moirai load, Clotho misses, Pixel sessions, standby worker count, first-frame time, p99 cognition latency, per-tier cost.

Initial triage (first 15 minutes)#

1. Acknowledge oshun-egbe-capacity-oncall.
2. Open the capacity surface and confirm v6.egbe.capacity.acknowledge_pressure is recorded.
3. Check active incidents and planned events: Commons festival, Aye campaign, district rollout, provider outage, or abuse wave.
4. Capture current values for Moirai load, Clotho misses, Pixel sessions, standby workers, p99 cognition latency, first-frame time, and fallback route share.
5. Confirm no competing broad verification or build process is exhausting the host before running local diagnostics.
6. Decide whether the fastest mitigation is fidelity shedding, scale-out, traffic shift to Tier-2 fallback, event throttling, or provider failover.

Diagnosis#

1. Determine the saturated resource:
   • Moirai CPU or queue pressure: cognition deadlines at risk.
   • Pixel Streaming workers: first-frame and session admission at risk.
   • World server shards: presence, AI perception, or district tick at risk.
   • Provider/model route: Clotho or Lachesis latency and cost at risk.
2. Compare current telemetry to the release gates: pnpm run verify:v6 latency-budgets, pnpm run verify:v6 commons-scale-load, pnpm run verify:v6 cost-budget, and pnpm run verify:v6 rendering-budgets.
3. Verify graceful degradation is attached before hard failure. If not, treat as SEV-2 even if users are not yet affected.
4. Check whether welfare or conduct signals are rising because agents are stuck in degraded cognition. If yes, alert Lilith welfare.

Mitigation#

1. Acknowledge mitigation in the Egbe Operator Console so the audit trail shows v6.egbe.capacity.acknowledge_pressure.
2. Shed fidelity in this order:
   • Move off-screen agents from Clotho/Lachesis to Atropos or behavior-tree fallback.
   • Reduce non-critical Chronicle rendering and background social simulation.
   • Route low-capability clients to Tier-2 fallback before Pixel sessions hard-cap.
3. Scale capacity:
   • Add Moirai workers for the affected region.
   • Add Pixel Streaming standby workers or shift admissions to a less-loaded POP.
   • Increase Redis stream consumers only after checking database pressure.
4. If an event is driving pressure, slow new shard admission, shrink festival crowd caps, or pause non-critical campaign windows.
5. If provider/model latency is the driver, apply provider failover and right-size model routes before increasing token budgets.
6. Communicate degraded mode honestly to players if fidelity shedding becomes visible.

Communication cadence#

Escalation#

• Platform capacity owner leads the incident.
• Pixel Streaming owner leads first-frame or worker-saturation issues.
• Moirai owner leads cognition misses and model-route pressure.
• Live ops owns event throttling and player messaging.
• Lilith welfare is paged if capacity degradation causes welfare signal spikes.

Recovery verification#

• Moirai load is below 85% and Clotho misses are below 2% for 60 minutes.
• Pixel sessions are below 90% capacity with at least 8 standby workers or the current regional equivalent.
• p99 cognition latency and first-frame budgets are back within release gate.
• pnpm run verify:v6 commons-scale-load, pnpm run verify:v6 cost-budget, and pnpm run verify:v6 latency-budgets pass against the incident telemetry or a documented post-incident replay.
• Any visible degraded mode is cleared or intentionally left in place with an owner and expiry.

Post-incident#

• Record peak load, time to mitigation, user impact, degraded-mode duration, scale actions, and audit refs.
• Update event capacity forecasts if the incident came from live programming.
• Update cost-budget thresholds if the model route was correctly chosen but the launch DAU projection was wrong.
• Add or update load tests when the incident exposes an untested pressure mode.
• Do not close until the operator surface, telemetry gate, and incident notes agree on the final state.