# Runbooks Index Operational playbooks for Oshun V1 and V6 launch-critical systems. Each runbook is owned by one team and lists its applicable services, the incident shape it handles, and the rough severity that should trigger it. When paging on-call, link directly to the relevant runbook below — don't ask the responder to hunt through `ls`. ## When to use this index - An alert fires and you don't know which runbook applies — search this table by service or incident type. - You're writing a new runbook — add a row here in the same PR. - You're auditing operational readiness — every domain should appear at least once. ## Index | Runbook | Owner | Applies to | Incident type | Severity | | -------------------------------------------------------------------------- | ----------------------- | --------------------------------------------------- | ---------------------------------------------------------------------- | --------------- | | [assistant-failure.md](assistant-failure.md) | iris team | iris/api, lilith/svc-ai | Assistant returns empty / errors at p95 | SEV-2 | | [database-backups.md](database-backups.md) | platform | every Postgres cluster | Take or restore a logical backup | n/a — scheduled | | [grounding-failure.md](grounding-failure.md) | psyche / iris | iris/api, psyche/grounded-answer | Grounded-answer system produces uncited claims | SEV-1 | | [ios-toolchain.md](ios-toolchain.md) | mobile | iris/mobile, arete/mobile, lilith/mobile, expo apps | Local or CI iOS build fails before producing an .app | n/a — setup | | [metis-disaster-recovery.md](metis-disaster-recovery.md) | UNASSIGNED | metis/service Postgres and metis-sources bucket | Restore the Metis estate: database, source objects, then reconcile | SEV-1 | | [model-workflow-rollback.md](model-workflow-rollback.md) | isis / nous | isis/generation-api, nous/\* providers | A model version produces regressions in eval pipeline | SEV-2 | | [moderation-surge.md](moderation-surge.md) | trust & safety | svc-moderation, content-\* | Sudden moderation queue spike (>2x baseline) | SEV-2 | | [oshun-v1.md](oshun-v1.md) | platform | full stack | V1 launch / cutover playbook | SEV-1 | | [persona-rollback.md](persona-rollback.md) | iris team | iris/api persona service | Persona update breaks live conversations | SEV-2 | | [postgres-client-migration.md](postgres-client-migration.md) | platform | every service still using raw `pg.Pool` | Migrate service to `@oshun/database` wrapper | n/a — refactor | | [privacy-incident.md](privacy-incident.md) | trust & safety | full stack | Suspected PII leak, unauthorized data access, GDPR escalation | SEV-1 | | [provenance-failure.md](provenance-failure.md) | veritas | veritas/\* services | Generated content provenance broken or missing | SEV-2 | | [provider-failover.md](provider-failover.md) | platform | lilith/svc-ai, isis/generation-api | Upstream model provider (OpenAI / Anthropic / etc.) is degraded | SEV-2 | | [v1-acceptance-matrix.md](v1-acceptance-matrix.md) | UNASSIGNED | V1 single-host stack | Not an incident — what a `gd:regional-loss` drill measured, per aspect | n/a — rehearsal | | [v1-disaster-recovery.md](v1-disaster-recovery.md) | prod-ops rota | V1 single-host stack | `gd:regional-loss` — host lost: rebuild from the off-box snapshot | SEV-1 | | [v1-dr-blob-store-loss.md](v1-dr-blob-store-loss.md) | workbench-platform rota | V1 MinIO | `gd:blob-store-loss` — rows survive, the bytes they point at do not | SEV-1 | | [v1-dr-bridge-host-loss.md](v1-dr-bridge-host-loss.md) | delivery-pipeline rota | V1 DCC bridge, enrolled hosts | `gd:bridge-host-loss` — the workstation the bridge runs on is gone | SEV-3 | | [v1-dr-database-loss.md](v1-dr-database-loss.md) | workbench-platform rota | V1 PostgreSQL | `gd:database-loss` — the relational store is unavailable | SEV-1 | | [v1-dr-model-provider-outage.md](v1-dr-model-provider-outage.md) | workbench-platform rota | V1 model providers | `gd:model-provider-outage` — the primary provider stops answering | SEV-2 | | [v1-dr-queue-loss.md](v1-dr-queue-loss.md) | delivery-pipeline rota | V1 generation-job queue and workers | `gd:queue-loss` — the queue loses its contents | SEV-2 | | [v1-dr-relational-corruption.md](v1-dr-relational-corruption.md) | workbench-platform rota | V1 PostgreSQL | `gd:relational-corruption` — the store answers with wrong rows | SEV-1 | | [v1-dr-search-index-loss.md](v1-dr-search-index-loss.md) | workbench-platform rota | V1 Elasticsearch | `gd:search-index-loss` — the search index is lost | SEV-2 | | [v1-dr-signing-key-compromise.md](v1-dr-signing-key-compromise.md) | delivery-pipeline rota | V1 C2PA signing, trust store | `gd:signing-key-compromise` — a signing key is disclosed | SEV-1 | | [v1-dr-signing-service-outage.md](v1-dr-signing-service-outage.md) | delivery-pipeline rota | V1 C2PA signer | `gd:signing-service-outage` — releases cannot be signed | SEV-2 | | [v1-follow-up-register.md](v1-follow-up-register.md) | UNASSIGNED | V1 single-host stack | Not an incident — what every game day found, who owns it, and by when | n/a — rehearsal | | [v1-game-day-plan.md](v1-game-day-plan.md) | prod-ops rota | V1 single-host stack | Not an incident — the plan a rehearsal of each scenario class follows | n/a — rehearsal | | [v1-observability-game-day.md](v1-observability-game-day.md) | prod-ops rota | V1 Hetzner stack and monitoring plane | Alert response and supervised detect→triage→rollback→recover rehearsal | SEV-1/2 | | [v1-reconciliation-and-replay.md](v1-reconciliation-and-replay.md) | prod-ops rota | V1 jobs, leases, events, outcomes, gates, releases | After any restore: fold back the state that accrued while it was down | SEV-1 | | [shell-outage.md](shell-outage.md) | platform | oshun-shell (the embedding shell) | Shell unavailable / cannot embed apps | SEV-1 | | [typecheck-cleanup.md](typecheck-cleanup.md) | shared-infra | per-library typecheck | Per-lib `tsc --noEmit` fails; root CI is green | n/a — debt | | [v6-agent-welfare-review.md](v6-agent-welfare-review.md) | trust & safety | egbe operator console, lilith-agent-welfare | V6 agent welfare thresholds require Lilith review | SEV-1/2/3 | | [v6-capacity-management.md](v6-capacity-management.md) | platform | moirai cluster, pxstream relay, world server | V6 cognition / Pixel Streaming capacity pressure | SEV-1/2/3 | | [v6-commons-moderation.md](v6-commons-moderation.md) | trust & safety | Commons Heart, realtime gateway, takedown executor | V6 Commons public-surface moderation or takedown review | SEV-1/2/3 | | [v6-incarnation-governance.md](v6-incarnation-governance.md) | aye bridge | aye bridge, ori service, threshold adapters | V6 Aye incarnation, return journal, or memory merge governance | SEV-1/2/3 | | [v6-production-setup.md](v6-production-setup.md) | platform | V6 service stack, web/admin, QA handoff | Stand up a new V6 production-like environment | n/a — setup | | [v6-steward-conduct-investigation.md](v6-steward-conduct-investigation.md) | trust & safety | egbe operator console, cognition audit log | V6 steward coercion / neglect / manipulation claim | SEV-1/2/3 | | [watermark-verification-failure.md](watermark-verification-failure.md) | aphrodite | aphrodite/stream-watermarking | Watermark verification fails on viewer playback | SEV-3 | ## Severity definitions - **SEV-1** — User-visible outage or data loss; page on-call immediately, page the secondary at the 15-minute mark. - **SEV-2** — Significant degradation (≥10% of users); page on-call. - **SEV-3** — Internal degradation or non-blocking regression; ticket during business hours. - **n/a** — Operational task that doesn't represent an active incident (setup, scheduled work, debt cleanup). ## Adding a new runbook 1. Create `docs/runbooks/.md` following the structure of an existing runbook (header with owner + last-reviewed; "When to use"; "Step-by-step"; "Common failures"). 2. Add a row to the table above in the same PR. 3. Cross-reference any new env vars or scripts in the runbook itself so future readers don't have to chase them down. 4. If the runbook covers a SEV-1 incident, also add it to the `docs/operations/on-call.md` cheat-sheet (once that doc lands).