# Runbook - V6 Capacity Management

> Use this runbook when the Moirai cluster, Pixel Streaming fleet, Tier-2
> fallback, or Commons event load approaches capacity limits. The goal is to
> degrade gracefully before cognition deadlines or first-frame budgets fail.

## Scope

Triggers:

- Egbe Operator Console surface `egbe-operator-surface-capacity-001` is degraded
  or critical.
- Audit action `v6.egbe.capacity.acknowledge_pressure` is required.
- Moirai load is above `85%`, Clotho misses are above `2%`, or Pixel Streaming
  sessions exceed the festival threshold.
- A Commons festival, Aye campaign, or district rollout increases active shards.
- `verify:v6 commons-scale-load`, `verify:v6 cost-budget`, or latency gates fail
  against current staging telemetry.

Applies to `egbe-moirai-cluster`, `egbe-world-server`, `egbe-realtime-gateway`,
`egbe-pxstream-relay`, Tier-2 fallback web, Pixel Streaming workers, and the V1
audit platform.

## Severity classification

| Sev   | Condition                                                                                                             |
| ----- | --------------------------------------------------------------------------------------------------------------------- |
| SEV-1 | Clotho misses exceed 5%, Pixel sessions hard-cap, or world shards stall for active players.                           |
| SEV-2 | Moirai load exceeds 85% for 15 minutes, Pixel sessions exceed 90% capacity, or fallback routing rises above forecast. |
| SEV-3 | A planned event is within 10% of capacity budget but no player-visible degradation has started.                       |

## Detection signals

- Audit action: `v6.egbe.capacity.acknowledge_pressure`.
- Capacity refs: `moirai-cluster:orun-region-01:pressure-sample`,
  `pixel-streaming-fleet:orun-region-01:session-sample`, and
  `capacity-plan:commons-festival-tier:gathering`.
- Governance refs: `moirai-policy:cognition-capacity-pressure`,
  `pixel-streaming-policy:fleet-saturation-response`, and
  `v6-policy:graceful-degradation-required`.
- Metrics: Moirai load, Clotho misses, Pixel sessions, standby worker count,
  first-frame time, p99 cognition latency, per-tier cost.

## Initial triage (first 15 minutes)

1. Acknowledge `oshun-egbe-capacity-oncall`.
2. Open the capacity surface and confirm `v6.egbe.capacity.acknowledge_pressure`
   is recorded.
3. Check active incidents and planned events: Commons festival, Aye campaign,
   district rollout, provider outage, or abuse wave.
4. Capture current values for Moirai load, Clotho misses, Pixel sessions,
   standby workers, p99 cognition latency, first-frame time, and fallback route
   share.
5. Confirm no competing broad verification or build process is exhausting the
   host before running local diagnostics.
6. Decide whether the fastest mitigation is fidelity shedding, scale-out,
   traffic shift to Tier-2 fallback, event throttling, or provider failover.

## Diagnosis

1. Determine the saturated resource:
   - Moirai CPU or queue pressure: cognition deadlines at risk.
   - Pixel Streaming workers: first-frame and session admission at risk.
   - World server shards: presence, AI perception, or district tick at risk.
   - Provider/model route: Clotho or Lachesis latency and cost at risk.
2. Compare current telemetry to the release gates:
   `pnpm run verify:v6 latency-budgets`,
   `pnpm run verify:v6 commons-scale-load`, `pnpm run verify:v6 cost-budget`,
   and `pnpm run verify:v6 rendering-budgets`.
3. Verify graceful degradation is attached before hard failure. If not, treat as
   SEV-2 even if users are not yet affected.
4. Check whether welfare or conduct signals are rising because agents are stuck
   in degraded cognition. If yes, alert Lilith welfare.

## Mitigation

1. Acknowledge mitigation in the Egbe Operator Console so the audit trail shows
   `v6.egbe.capacity.acknowledge_pressure`.
2. Shed fidelity in this order:
   - Move off-screen agents from Clotho/Lachesis to Atropos or behavior-tree
     fallback.
   - Reduce non-critical Chronicle rendering and background social simulation.
   - Route low-capability clients to Tier-2 fallback before Pixel sessions
     hard-cap.
3. Scale capacity:
   - Add Moirai workers for the affected region.
   - Add Pixel Streaming standby workers or shift admissions to a less-loaded
     POP.
   - Increase Redis stream consumers only after checking database pressure.
4. If an event is driving pressure, slow new shard admission, shrink festival
   crowd caps, or pause non-critical campaign windows.
5. If provider/model latency is the driver, apply provider failover and
   right-size model routes before increasing token budgets.
6. Communicate degraded mode honestly to players if fidelity shedding becomes
   visible.

## Communication cadence

| Phase           | Audience           | Cadence          | Content                                                       |
| --------------- | ------------------ | ---------------- | ------------------------------------------------------------- |
| Acknowledgement | Capacity on-call   | Within 5 minutes | Region, saturated resource, player impact, first mitigation.  |
| Active incident | Platform lead      | Every 15 minutes | Load, queue age, Clotho misses, Pixel sessions, scale state.  |
| SEV-1 update    | Leadership/support | Every 30 minutes | Customer impact, event throttles, status-page recommendation. |
| Closure         | Incident channel   | On closure       | Final metrics, user impact, audit refs, follow-up owners.     |

## Escalation

- Platform capacity owner leads the incident.
- Pixel Streaming owner leads first-frame or worker-saturation issues.
- Moirai owner leads cognition misses and model-route pressure.
- Live ops owns event throttling and player messaging.
- Lilith welfare is paged if capacity degradation causes welfare signal spikes.

## Recovery verification

- Moirai load is below 85% and Clotho misses are below 2% for 60 minutes.
- Pixel sessions are below 90% capacity with at least 8 standby workers or the
  current regional equivalent.
- p99 cognition latency and first-frame budgets are back within release gate.
- `pnpm run verify:v6 commons-scale-load`, `pnpm run verify:v6 cost-budget`, and
  `pnpm run verify:v6 latency-budgets` pass against the incident telemetry or a
  documented post-incident replay.
- Any visible degraded mode is cleared or intentionally left in place with an
  owner and expiry.

## Post-incident

- Record peak load, time to mitigation, user impact, degraded-mode duration,
  scale actions, and audit refs.
- Update event capacity forecasts if the incident came from live programming.
- Update cost-budget thresholds if the model route was correctly chosen but the
  launch DAU projection was wrong.
- Add or update load tests when the incident exposes an untested pressure mode.
- Do not close until the operator surface, telemetry gate, and incident notes
  agree on the final state.