# Runbook — Moderation Surge

> Trust & Safety review/moderation/appeal queues spike beyond surge-rota
> capacity; backlog risks SLA breach on safety-critical decisions. Use to raise
> queue priority, freeze risky publication, route surge rota, and communicate
> operator status.

## Scope

Triggers:

- `queue_review` or `queue_moderation` depth above SLO for >30 min.
- `queue_appeal` aging beyond the 48-hour SLA.
- A coordinated abuse incident (mass reports, prompt-injection wave, scraping
  wave, brigading) targets the platform.
- A model promotion or persona promotion causes a false-positive flood that
  swamps reviewers.
- An external event (news cycle, holiday, crisis) raises Tara/Veritas reviewer
  load above baseline.

## Severity classification

| Sev  | Condition                                                                                                                               |
| ---- | --------------------------------------------------------------------------------------------------------------------------------------- |
| sev1 | `queue_review` contains items tagged `crisis` or `csam_suspected` not actioned within 1 hour, OR appeals backlog crosses regulator SLA. |
| sev2 | `queue_moderation` depth > 2× SLO sustained for >30 min; appeals SLA breached for any cohort.                                           |
| sev3 | Single sub-queue (e.g., persona-appeals only) backed up; no customer-visible publication risk.                                          |

Source events: `admin.queue.depth_sampled`, `admin.review.decision_recorded`,
`admin.appeal.opened`, `admin.appeal.aged_out` from `admin-taxonomy.ts`. Queue
health targets live in `dashboards-queue-health.ts`.

## Detection signals

- Alerts: queue-depth alerts from `dashboards-queue-health.ts` for
  `queue_review`, `queue_moderation`, `queue_appeal`.
- Operator signals: T&S lead receives an SLA-breach digest hourly during surge.
- Support correlate: spike in tickets tagged
  `support.case.tags.account_action_disputed` or
  `support.case.tags.appeal_status`.

## Initial triage (first 15 minutes)

1. Acknowledge `oshun-trust-safety-oncall`.
2. Open `dashboards-queue-health.ts` — record depth, SLA breach count, and the
   per-policy-class breakdown.
3. Identify the leading policy class driving the depth.
4. Decide root cause hypothesis:
   - Reviewer capacity drop (rota shortage / holiday) → activate surge rota.
   - Model false-positive flood → suspect the most recent Lilith promotion;
     prepare rollback via `assistant-failure.md` policy path.
   - External event → coordinate with launch-support team on public comms.
   - Abuse wave → coordinate with security on detection rules.
5. **Crisis filter first**: regardless of root cause, items tagged `crisis`,
   `self_harm`, `csam_suspected`, `imminent_violence` skip the queue and route
   directly to the on-call senior reviewer. Verify the crisis-router is healthy
   before any other action.

## Mitigation

1. **Raise queue priority** for safety-critical classes via the admin
   queue-priority hook (admin-taxonomy event `admin.queue.priority_raised`).
   Lower-class items deprioritized in the same operation.
2. **Surge rota activation**: page the surge-reviewer rota per the T&S schedule.
   Verify each surge reviewer has refreshed RBAC and policy training within the
   last 90 days; rotate any who have not.
3. **Freeze risky publication**: enable the publish-freeze flag for the affected
   policy class. Publish gates emit `studio.publish.frozen` events; authors see
   the `studio.publish.frozen_banner_shown` event-driven banner.
4. **Model-driven false-positive flood**: revert the implicated Lilith policy
   version or model version per `model-workflow-rollback.md`. Do not skip review
   on items already enqueued — they remain reviewable but the intake rate drops
   to baseline.
5. **Abuse wave**: coordinate with security to enable upstream rate limits and
   abuse-signature filters. Add the abuse cohort to the moderation priority
   queue so cohort-wide actions can be taken in bulk where policy permits.
6. **Operator status**: publish the `incident.internal.slack.investigating`
   template to the T&S Slack with queue depth, SLA breach count, surge plan, and
   projected clear-by ETA. Refresh hourly until clear.

## Communication cadence

Moderation surge is rarely a customer-visible incident unless publication
freezes hit launch-quality content. If publication is frozen on a public domain
(Tara, Veritas, Nyx), publish:

| Phase                                 | Template ID                              | Audience    |
| ------------------------------------- | ---------------------------------------- | ----------- |
| Status (publication freeze in effect) | `incident.customer.in_app.investigating` | customer    |
| Resolved (freeze lifted)              | `incident.customer.status_page.resolved` | status_page |
| Internal (every hour during surge)    | `incident.internal.slack.investigating`  | internal    |
| Leadership (sev1 only)                | `incident.leadership.email.resolved`     | leadership  |

For regulator-reportable events (CSAM, imminent violence), route through the
compliance lead — the `incident.regulator.email.resolved` template applies once
the incident is closed.

## Escalation

- T&S lead is the incident commander for moderation surges.
- For sev1, page the privacy lead in parallel — any account action against the
  surge cohort needs privacy review.
- For abuse-wave sev1, page the security lead.
- Standard 5/5/15/30/60 escalation otherwise applies for reviewer-rota paging.

## Recovery verification

- `queue_review` and `queue_moderation` depth back within SLO for 60 min.
- No new SLA breaches for any sub-queue for 60 min.
- Crisis-router success rate ≥ 99.9% for the surge window.
- Surge rota members offboarded back to baseline; the surge log captures
  decisions and reviewer assignments for the audit trail.
- If publish-freeze was active, the lifted freeze passes a smoke check of 20
  freshly-published items.

## Post-incident

- Postmortem must include: queue-depth timeline, reviewer headcount over time,
  root-cause hypothesis confirmation, model/persona/policy rollback if any,
  customer-visible publication-freeze duration, and any regulator-reportable
  items.
- Update reviewer training (V1-OPS-training) if the surge revealed a
  decision-gap.
- Review the appeals backlog separately — appeals that aged beyond SLA must
  individually clear with a documented decision.
- File an abuse-pattern note with security if the surge was abuse-driven.