Disciplines · Runbooks

Runbooks Index

service or incident type.

4sections6 minread

On this page

Operational playbooks for Oshun V1 and V6 launch-critical systems. Each runbook is owned by one team and lists its applicable services, the incident shape it handles, and the rough severity that should trigger it. When paging on-call, link directly to the relevant runbook below — don't ask the responder to hunt through ls.

When to use this index#

  • An alert fires and you don't know which runbook applies — search this table by service or incident type.
  • You're writing a new runbook — add a row here in the same PR.
  • You're auditing operational readiness — every domain should appear at least once.

Index#

Runbook Owner Applies to Incident type Severity
assistant-failure.md iris team iris/api, lilith/svc-ai Assistant returns empty / errors at p95 SEV-2
database-backups.md platform every Postgres cluster Take or restore a logical backup n/a — scheduled
grounding-failure.md psyche / iris iris/api, psyche/grounded-answer Grounded-answer system produces uncited claims SEV-1
ios-toolchain.md mobile iris/mobile, arete/mobile, lilith/mobile, expo apps Local or CI iOS build fails before producing an .app n/a — setup
metis-disaster-recovery.md UNASSIGNED metis/service Postgres and metis-sources bucket Restore the Metis estate: database, source objects, then reconcile SEV-1
model-workflow-rollback.md isis / nous isis/generation-api, nous/* providers A model version produces regressions in eval pipeline SEV-2
moderation-surge.md trust & safety svc-moderation, content-* Sudden moderation queue spike (>2x baseline) SEV-2
oshun-v1.md platform full stack V1 launch / cutover playbook SEV-1
persona-rollback.md iris team iris/api persona service Persona update breaks live conversations SEV-2
postgres-client-migration.md platform every service still using raw pg.Pool Migrate service to @oshun/database wrapper n/a — refactor
privacy-incident.md trust & safety full stack Suspected PII leak, unauthorized data access, GDPR escalation SEV-1
provenance-failure.md veritas veritas/* services Generated content provenance broken or missing SEV-2
provider-failover.md platform lilith/svc-ai, isis/generation-api Upstream model provider (OpenAI / Anthropic / etc.) is degraded SEV-2
v1-acceptance-matrix.md UNASSIGNED V1 single-host stack Not an incident — what a gd:regional-loss drill measured, per aspect n/a — rehearsal
v1-disaster-recovery.md prod-ops rota V1 single-host stack gd:regional-loss — host lost: rebuild from the off-box snapshot SEV-1
v1-dr-blob-store-loss.md workbench-platform rota V1 MinIO gd:blob-store-loss — rows survive, the bytes they point at do not SEV-1
v1-dr-bridge-host-loss.md delivery-pipeline rota V1 DCC bridge, enrolled hosts gd:bridge-host-loss — the workstation the bridge runs on is gone SEV-3
v1-dr-database-loss.md workbench-platform rota V1 PostgreSQL gd:database-loss — the relational store is unavailable SEV-1
v1-dr-model-provider-outage.md workbench-platform rota V1 model providers gd:model-provider-outage — the primary provider stops answering SEV-2
v1-dr-queue-loss.md delivery-pipeline rota V1 generation-job queue and workers gd:queue-loss — the queue loses its contents SEV-2
v1-dr-relational-corruption.md workbench-platform rota V1 PostgreSQL gd:relational-corruption — the store answers with wrong rows SEV-1
v1-dr-search-index-loss.md workbench-platform rota V1 Elasticsearch gd:search-index-loss — the search index is lost SEV-2
v1-dr-signing-key-compromise.md delivery-pipeline rota V1 C2PA signing, trust store gd:signing-key-compromise — a signing key is disclosed SEV-1
v1-dr-signing-service-outage.md delivery-pipeline rota V1 C2PA signer gd:signing-service-outage — releases cannot be signed SEV-2
v1-follow-up-register.md UNASSIGNED V1 single-host stack Not an incident — what every game day found, who owns it, and by when n/a — rehearsal
v1-game-day-plan.md prod-ops rota V1 single-host stack Not an incident — the plan a rehearsal of each scenario class follows n/a — rehearsal
v1-observability-game-day.md prod-ops rota V1 Hetzner stack and monitoring plane Alert response and supervised detect→triage→rollback→recover rehearsal SEV-1/2
v1-reconciliation-and-replay.md prod-ops rota V1 jobs, leases, events, outcomes, gates, releases After any restore: fold back the state that accrued while it was down SEV-1
shell-outage.md platform oshun-shell (the embedding shell) Shell unavailable / cannot embed apps SEV-1
typecheck-cleanup.md shared-infra per-library typecheck Per-lib tsc --noEmit fails; root CI is green n/a — debt
v6-agent-welfare-review.md trust & safety egbe operator console, lilith-agent-welfare V6 agent welfare thresholds require Lilith review SEV-1/2/3
v6-capacity-management.md platform moirai cluster, pxstream relay, world server V6 cognition / Pixel Streaming capacity pressure SEV-1/2/3
v6-commons-moderation.md trust & safety Commons Heart, realtime gateway, takedown executor V6 Commons public-surface moderation or takedown review SEV-1/2/3
v6-incarnation-governance.md aye bridge aye bridge, ori service, threshold adapters V6 Aye incarnation, return journal, or memory merge governance SEV-1/2/3
v6-production-setup.md platform V6 service stack, web/admin, QA handoff Stand up a new V6 production-like environment n/a — setup
v6-steward-conduct-investigation.md trust & safety egbe operator console, cognition audit log V6 steward coercion / neglect / manipulation claim SEV-1/2/3
watermark-verification-failure.md aphrodite aphrodite/stream-watermarking Watermark verification fails on viewer playback SEV-3

Severity definitions#

  • SEV-1 — User-visible outage or data loss; page on-call immediately, page the secondary at the 15-minute mark.
  • SEV-2 — Significant degradation (≥10% of users); page on-call.
  • SEV-3 — Internal degradation or non-blocking regression; ticket during business hours.
  • n/a — Operational task that doesn't represent an active incident (setup, scheduled work, debt cleanup).

Adding a new runbook#

  1. Create docs/runbooks/<short-name>.md following the structure of an existing runbook (header with owner + last-reviewed; "When to use"; "Step-by-step"; "Common failures").
  2. Add a row to the table above in the same PR.
  3. Cross-reference any new env vars or scripts in the runbook itself so future readers don't have to chase them down.
  4. If the runbook covers a SEV-1 incident, also add it to the docs/operations/on-call.md cheat-sheet (once that doc lands).