# Runbook — `<SUBSYSTEM-NAME>`

**Status:** `Draft | Active | Deprecated` &nbsp;**Date:** `<YYYY-MM-DD>`
&nbsp;**Owner:** `eng-<domain>` &nbsp;**Backlog item:** `<V1-P2-NNNN>`
&nbsp;**Audience:** on-call SRE, incident commander, domain leads

> This file is the V1-P2-0372 runbook **template**. Every operationally-owned
> subsystem in `docs/releases/v1/specs/todos-p2.md` Part III (Tier-A
> replacements) must own a populated copy at
> `docs/<domain>/runbooks/<subsystem>.md`. The header above is mandatory — if
> any of `Status`, `Date`, `Owner`, `Backlog item`, `Audience` is missing the
> runbook is not considered complete. Fill every `[FILL IN]` block; empty
> placeholders are not acceptable. When you fill the file in, replace this
> blockquote with a one-paragraph summary of the subsystem.

## 1. Subsystem at a glance

- **Purpose.** [FILL IN: one sentence describing what this subsystem does for
  the platform. e.g. "Issues short-lived OAuth tokens for vendor APIs used by
  Lilith and Yemaya."]
- **Code locations.** [FILL IN: every `libs/<x>/...` and `apps/<x>/...` path
  that constitutes this subsystem. Be exhaustive — readers use this to grep.]
- **Owning team.** [FILL IN: Slack `#<channel>`, GitHub team `@<org>/<team>`,
  rotating on-call schedule URL.]
- **Upstream consumers.** [FILL IN: every domain or service that calls this
  subsystem — used to scope blast radius during incidents.]
- **Downstream dependencies.** [FILL IN: every external API, internal service,
  or shared infrastructure component this subsystem talks to. See §3 (rate
  limits) and §6 (vendor SLA) for the contract details.]

## 2. SLOs and error budgets

State the user-facing service-level objectives in absolute terms (latency
percentile, success rate, freshness window) and link to the dashboard panel that
displays them.

| Metric       | SLO                     | Window      | Burn-rate alert | Dashboard panel                                                                              |
| ------------ | ----------------------- | ----------- | --------------- | -------------------------------------------------------------------------------------------- |
| Availability | [FILL IN: e.g. 99.5%]   | rolling 30d | 14.4× over 1h   | [FILL IN: link to `docker/observability/grafana/dashboards/<domain>/<subsystem>.json` panel] |
| p95 latency  | [FILL IN: e.g. ≤ 750ms] | rolling 7d  | 14.4× over 1h   | [FILL IN]                                                                                    |
| Error rate   | [FILL IN: e.g. < 0.5%]  | rolling 1h  | 6× over 5m      | [FILL IN]                                                                                    |

**Error budget policy.** [FILL IN: e.g. "If the 30-day budget is consumed, no
non-critical changes ship until budget recovers above 50%. Hot-fixes require IC
sign-off."]

## 3. Rate limits

Every external API and internal service has a rate limit. **Document the hard
ceiling, the soft ceiling, who hits it first, and how the subsystem behaves at
the ceiling.**

| Direction | Counterparty                              | Hard limit                                                       | Soft limit                 | Bucket key                  | Behaviour at limit                                                                                     |
| --------- | ----------------------------------------- | ---------------------------------------------------------------- | -------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------ |
| Outbound  | [FILL IN: e.g. OpenAI `chat.completions`] | [FILL IN: e.g. 10k req/min, 90M tok/min, per org]                | [FILL IN: e.g. 8k req/min] | [FILL IN: tenant or global] | [FILL IN: e.g. "circuit breaker opens; failover-manager (libs/iris/failover) routes to next provider"] |
| Outbound  | [FILL IN]                                 | [FILL IN]                                                        | [FILL IN]                  | [FILL IN]                   | [FILL IN]                                                                                              |
| Inbound   | [FILL IN: e.g. external tenants]          | [FILL IN: e.g. 1000 req/min per tenant, enforced at API gateway] | [FILL IN]                  | [FILL IN]                   | [FILL IN: e.g. "HTTP 429 with `Retry-After`; SLO error rate excludes 429s"]                            |

**Source of truth.**

- Outbound limits live in [FILL IN: e.g. `libs/<domain>/<subsystem>/config.ts`]
  and are enforced by `@oshun/http-client` retry + circuit-breaker config.
- Inbound limits live in [FILL IN: e.g. the API gateway middleware at
  `apps/<domain>/api/src/middleware/rate-limit.ts`].
- Vendor-side limits are tracked in §6 against the contractual SLA.

**Verification.** Load test at 1.5× the soft limit; confirm the circuit breaker
opens and the failover path engages without leaking 5xx to the caller. Capture a
screenshot of the dashboard during the test and link it in the rollout PR.

## 4. Outage drill

A drill is a planned, scheduled exercise where the on-call team practises
recovery from a realistic failure, **with traffic in production or staging under
controlled conditions**. Drills are mandatory for every Tier-A subsystem, run at
least once per quarter.

### 4.1 Drill scenarios

The drill must cover every failure mode you'd page on. At minimum:

1. **Vendor outage.** [FILL IN: "Block egress to <vendor> at the network layer
   for 10 minutes during off-peak. Verify failover-manager routes to the
   secondary provider and the SLO panel does not breach the burn-rate alert."]
2. **Database failover.** [FILL IN: "Trigger a manual primary-replica failover
   on the `<domain>_db` PostgreSQL instance. Verify the BFF / API layer
   reconnects within 30s, that no data is lost, and that the
   subscriber-projection backlog clears within 5m."]
3. **Cache stampede.** [FILL IN: e.g. "Flush the Redis cache for the subsystem
   and observe whether downstream amplification triggers circuit-breakers or an
   LLM cost spike."]
4. **Quota exhaustion.** [FILL IN: e.g. "Force a single tenant to exceed the
   cost-tracking quota documented in
   `docs/releases/p2/isis-cost-tracking-contract.md`. Verify the quota
   middleware refuses the request before the provider call leaves the gateway."]
5. **Subscriber lag.** [FILL IN: e.g. "Pause the `<subsystem>-projector`
   consumer for 30 minutes; verify the alert fires and the projector recovers
   without duplicate processing once resumed (idempotency key = `event.id`)."]

### 4.2 Drill checklist

For every drill:

- [ ] Schedule announced ≥48h ahead in `#oncall-<domain>` and on the status
      page.
- [ ] Pre-drill snapshot of the dashboard captured.
- [ ] Drill IC and observer roles assigned (different humans).
- [ ] Drill runs with a timer; abort criteria stated upfront.
- [ ] Post-drill blameless retro within 5 business days; gaps land as backlog
      tickets in the next sprint.

### 4.3 Drill frequency

[FILL IN: typically Q1/Q3 for vendor outage + DB failover, Q2/Q4 for quota +
subscriber-lag. Reference the calendar in `<runbook-link>`.]

## 5. Escalation contacts

Three-tier escalation. Use the lowest tier first; only escalate when the tier
above hasn't engaged within the window.

| Tier | Trigger                                                   | Who                                       | How                                                         | SLA         |
| ---- | --------------------------------------------------------- | ----------------------------------------- | ----------------------------------------------------------- | ----------- |
| 0    | Anomaly, no user impact                                   | Domain on-call                            | PagerDuty schedule [FILL IN: schedule URL]                  | Best-effort |
| 1    | SLO burn-rate alert, ≤25% blast radius                    | Domain IC                                 | PagerDuty escalation policy [FILL IN: policy URL]           | 15m         |
| 2    | SLO breach OR ≥25% blast radius OR vendor SLA breach      | Domain lead + Platform IC + Comms         | PagerDuty + Slack `#incident-<domain>` + status-page update | 5m          |
| 3    | Cross-domain incident OR data-loss risk OR security event | VP Engineering + Security on-call + Legal | PagerDuty + executive bridge                                | Immediate   |

**Specific contacts.** Always cross-check with the on-call rotation at [FILL IN:
schedule URL] — names rotate.

| Role                   | Who (current)                                           | Backup    | Slack     | PagerDuty |
| ---------------------- | ------------------------------------------------------- | --------- | --------- | --------- |
| Domain on-call         | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |
| Domain IC              | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |
| Domain lead            | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |
| Platform IC            | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |
| Vendor-account contact | [FILL IN: e.g. OpenAI TAM email + business-hours phone] | [FILL IN] | n/a       | n/a       |
| Security on-call       | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |
| Comms / status page    | [FILL IN]                                               | [FILL IN] | [FILL IN] | [FILL IN] |

## 6. Vendor SLA tracking

Every external vendor in §3's outbound list has a contractual SLA. Track each
one against the subsystem's own SLOs so a vendor-side breach is detected and
credit is claimed within the contractual window.

| Vendor    | Tier                               | Uptime SLA       | Latency SLA              | Credit window  | Credit %          | Contract URL                                       | Account contact |
| --------- | ---------------------------------- | ---------------- | ------------------------ | -------------- | ----------------- | -------------------------------------------------- | --------------- |
| [FILL IN] | [FILL IN: Enterprise / Pro / Free] | [FILL IN: 99.9%] | [FILL IN: e.g. p95 < 2s] | [FILL IN: 30d] | [FILL IN: 10–30%] | [FILL IN: link to signed contract in Drive/Notion] | [FILL IN]       |

**Detection.** Each vendor's availability is sampled by the `@oshun/http-client`
request metrics — see the [architecture-overview](./architecture-overview.md)
for the trace path. The Grafana panel at [FILL IN: panel link] surfaces the
per-vendor success rate over the credit window.

**Claiming credit.** When the per-vendor success rate falls below the SLA
threshold, file the support ticket within the credit window. The ticket template
lives at [FILL IN: e.g. `docs/<domain>/runbooks/vendor-sla-claim-template.md`].

**Vendor-side rate-limit changes.** Subscribe to the vendor's status page and
changelog. Document the channels in the table below — when a vendor silently
raises a rate limit, the soft ceiling in §3 must be updated in the same PR that
lands the new behaviour.

## 7. Cost guardrails

The §2.6 quality bar requires cost-aware operation. This section defines the
tenant-level and platform-level guardrails for this subsystem.

### 7.1 Per-tenant quotas

Per-tenant cost ceilings flow through the contract documented in
[`isis-cost-tracking-contract.md`](./isis-cost-tracking-contract.md)
(V1-P2-0005). For this subsystem:

| Tenant tier | Daily cap             | Monthly cap   | Behaviour at cap                         | Override path          |
| ----------- | --------------------- | ------------- | ---------------------------------------- | ---------------------- |
| Free        | [FILL IN: e.g. $0.50] | [FILL IN: $5] | Reject with 402 + `X-Quota-Reset` header | Self-service upgrade   |
| Pro         | [FILL IN]             | [FILL IN]     | Throttle to 50% rate, alert account      | Account team           |
| Enterprise  | [FILL IN]             | [FILL IN]     | Soft alert at 80%, IC ack at 100%        | Contract renegotiation |

**Quota observability.** Grafana panel [FILL IN: panel link] shows the top-N
tenants by burn rate over the previous 24h. The
[isis/generation-pipeline.json](../../../docker/observability/grafana/dashboards/isis/generation-pipeline.json)
dashboard's "cost-quota burn" panel is the reference shape.

### 7.2 Platform-level cost ceiling

A daily platform-wide cap protects against quota-bypass bugs and runaway cron
loops. **The ceiling here is the maximum the subsystem can spend in 24h before
paging.**

| Period | Soft ceiling         | Hard ceiling          | Page-on-breach |
| ------ | -------------------- | --------------------- | -------------- |
| Hour   | [FILL IN: e.g. $50]  | [FILL IN: e.g. $150]  | Tier-1         |
| Day    | [FILL IN: e.g. $500] | [FILL IN: e.g. $1500] | Tier-2         |

**Hard-ceiling behaviour.** [FILL IN: e.g. "All outbound provider calls are
short-circuited at the gateway with `503 quota_exhausted`. Existing in-flight
requests are allowed to drain. Recovery is manual: ack the incident, raise the
ceiling, document the new limit here in the same PR that closes the incident."]

### 7.3 Scheduled cost reviews

Every Tier-A subsystem is reviewed weekly for cost trend; monthly for quota cap
accuracy.

- Weekly review: [FILL IN: meeting time, link to retro doc, dashboard URL].
- Monthly review: [FILL IN: meeting time, attendee list, agenda doc].
- Cost-anomaly alert: [FILL IN: PagerDuty service + threshold, e.g. "+30% WoW
  for any single tenant, page domain on-call within 5m"].

## 8. Deployment + rollback

Reference [FILL IN: e.g. `docs/<domain>/runbooks/deployment.md`] for the deploy
procedure. The rollback path **must** be documented here even if the deploy
procedure lives elsewhere.

| Step | Action                                                                                                | Verify    | Owner   |
| ---- | ----------------------------------------------------------------------------------------------------- | --------- | ------- |
| 1    | [FILL IN: e.g. "Pause the BullMQ `<queue>` workers via `pnpm nx run <subsystem>:scale --replicas=0`"] | [FILL IN] | On-call |
| 2    | [FILL IN: e.g. "Roll back deployment to the previous tagged release"]                                 | [FILL IN] | On-call |
| 3    | [FILL IN: e.g. "Replay missed events from the Redis Stream up to the rollback timestamp"]             | [FILL IN] | On-call |
| 4    | [FILL IN: e.g. "Resume workers; confirm SLO panels green"]                                            | [FILL IN] | On-call |

## 9. Common alerts and first-response

For each alert that pages this subsystem, document the runbook entry the alert
links to. **An alert without a runbook entry should not be on the on-call
rotation.**

| Alert                                   | Severity  | Runbook anchor                 | First-response                                                                                                                                                   |
| --------------------------------------- | --------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [FILL IN: e.g. `SLOAvailability1hBurn`] | Tier-1    | [FILL IN: link to a §10 entry] | [FILL IN: e.g. "Open `oshun/platform-overview.json`, check the per-vendor success-rate panel; if a single vendor is at 0%, run §4.1 scenario 1 (vendor outage)"] |
| [FILL IN]                               | [FILL IN] | [FILL IN]                      | [FILL IN]                                                                                                                                                        |

## 10. Diagnostic playbooks

One subsection per recurring issue. Replace each `## 10.x [FILL IN]` heading
with the symptom; below it document the steps to confirm the cause and the
action to take.

### 10.1 [FILL IN: symptom]

**Confirm cause.** [FILL IN: queries, panel checks, log greps.]

**Action.** [FILL IN: explicit commands.]

**Verify recovery.** [FILL IN: SLO panel returning to green, queue depth
draining, etc.]

### 10.2 [FILL IN: symptom]

…

## 11. Verification — runbook completeness

Before the runbook is marked Active, the domain owner must verify:

- [ ] Every `[FILL IN]` placeholder above has been replaced with a concrete
      value (a string, a link, or `n/a` with a reason).
- [ ] Every dashboard URL resolves and the linked panel exists.
- [ ] Every PagerDuty link resolves to a non-empty schedule or policy.
- [ ] Every vendor entry in §6 has a contract URL (drive/notion/etc.) that a
      reviewer can click.
- [ ] Every alert in §9 maps to a §10 playbook anchor.
- [ ] An outage drill (§4) has been executed at least once and the retro is
      linked.
- [ ] The §3 rate limits have been load-tested at 1.5× the soft ceiling and the
      dashboard screenshot is linked in the verification PR.

A runbook that fails any of these checks is `Status: Draft` and cannot be
referenced as an authoritative source during an incident.

## 12. Change log

When updating the runbook:

1. Increment `Date` in the header.
2. Append an entry below.
3. Land the change in a PR that another domain on-call has reviewed.

| Date           | Author            | Change                                                                 |
| -------------- | ----------------- | ---------------------------------------------------------------------- |
| `<YYYY-MM-DD>` | `<github handle>` | Initial fill from `docs/releases/p2/runbook-template.md` (V1-P2-0372). |

---

_This template is the source of truth for runbook structure. When the template
changes, every existing runbook is reviewed within 30 days for sections that
need backfill. The change-log row in §12 is the audit trail._
