Status: Draft | Active | Deprecated Date: <YYYY-MM-DD>
Owner: eng-<domain> Backlog item: <V1-P2-NNNN>
Audience: on-call SRE, incident commander, domain leads
This file is the V1-P2-0372 runbook template. Every operationally-owned subsystem in
docs/releases/v1/specs/todos-p2.mdPart III (Tier-A replacements) must own a populated copy atdocs/<domain>/runbooks/<subsystem>.md. The header above is mandatory — if any ofStatus,Date,Owner,Backlog item,Audienceis missing the runbook is not considered complete. Fill every[FILL IN]block; empty placeholders are not acceptable. When you fill the file in, replace this blockquote with a one-paragraph summary of the subsystem.
1. Subsystem at a glance#
- Purpose. [FILL IN: one sentence describing what this subsystem does for the platform. e.g. "Issues short-lived OAuth tokens for vendor APIs used by Lilith and Yemaya."]
- Code locations. [FILL IN: every
libs/<x>/...andapps/<x>/...path that constitutes this subsystem. Be exhaustive — readers use this to grep.] - Owning team. [FILL IN: Slack
#<channel>, GitHub team@<org>/<team>, rotating on-call schedule URL.] - Upstream consumers. [FILL IN: every domain or service that calls this subsystem — used to scope blast radius during incidents.]
- Downstream dependencies. [FILL IN: every external API, internal service, or shared infrastructure component this subsystem talks to. See §3 (rate limits) and §6 (vendor SLA) for the contract details.]
2. SLOs and error budgets#
State the user-facing service-level objectives in absolute terms (latency percentile, success rate, freshness window) and link to the dashboard panel that displays them.
| Metric | SLO | Window | Burn-rate alert | Dashboard panel |
|---|---|---|---|---|
| Availability | [FILL IN: e.g. 99.5%] | rolling 30d | 14.4× over 1h | [FILL IN: link to docker/observability/grafana/dashboards/<domain>/<subsystem>.json panel] |
| p95 latency | [FILL IN: e.g. ≤ 750ms] | rolling 7d | 14.4× over 1h | [FILL IN] |
| Error rate | [FILL IN: e.g. < 0.5%] | rolling 1h | 6× over 5m | [FILL IN] |
Error budget policy. [FILL IN: e.g. "If the 30-day budget is consumed, no non-critical changes ship until budget recovers above 50%. Hot-fixes require IC sign-off."]
3. Rate limits#
Every external API and internal service has a rate limit. Document the hard ceiling, the soft ceiling, who hits it first, and how the subsystem behaves at the ceiling.
| Direction | Counterparty | Hard limit | Soft limit | Bucket key | Behaviour at limit |
|---|---|---|---|---|---|
| Outbound | [FILL IN: e.g. OpenAI chat.completions] |
[FILL IN: e.g. 10k req/min, 90M tok/min, per org] | [FILL IN: e.g. 8k req/min] | [FILL IN: tenant or global] | [FILL IN: e.g. "circuit breaker opens; failover-manager (libs/iris/failover) routes to next provider"] |
| Outbound | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Inbound | [FILL IN: e.g. external tenants] | [FILL IN: e.g. 1000 req/min per tenant, enforced at API gateway] | [FILL IN] | [FILL IN] | [FILL IN: e.g. "HTTP 429 with Retry-After; SLO error rate excludes 429s"] |
Source of truth.
- Outbound limits live in [FILL IN: e.g.
libs/<domain>/<subsystem>/config.ts] and are enforced by@oshun/http-clientretry + circuit-breaker config. - Inbound limits live in [FILL IN: e.g. the API gateway middleware at
apps/<domain>/api/src/middleware/rate-limit.ts]. - Vendor-side limits are tracked in §6 against the contractual SLA.
Verification. Load test at 1.5× the soft limit; confirm the circuit breaker opens and the failover path engages without leaking 5xx to the caller. Capture a screenshot of the dashboard during the test and link it in the rollout PR.
4. Outage drill#
A drill is a planned, scheduled exercise where the on-call team practises recovery from a realistic failure, with traffic in production or staging under controlled conditions. Drills are mandatory for every Tier-A subsystem, run at least once per quarter.
4.1 Drill scenarios#
The drill must cover every failure mode you'd page on. At minimum:
- Vendor outage. [FILL IN: "Block egress to <vendor> at the network layer for 10 minutes during off-peak. Verify failover-manager routes to the secondary provider and the SLO panel does not breach the burn-rate alert."]
- Database failover. [FILL IN: "Trigger a manual primary-replica failover
on the
<domain>_dbPostgreSQL instance. Verify the BFF / API layer reconnects within 30s, that no data is lost, and that the subscriber-projection backlog clears within 5m."] - Cache stampede. [FILL IN: e.g. "Flush the Redis cache for the subsystem and observe whether downstream amplification triggers circuit-breakers or an LLM cost spike."]
- Quota exhaustion. [FILL IN: e.g. "Force a single tenant to exceed the
cost-tracking quota documented in
docs/releases/p2/isis-cost-tracking-contract.md. Verify the quota middleware refuses the request before the provider call leaves the gateway."] - Subscriber lag. [FILL IN: e.g. "Pause the
<subsystem>-projectorconsumer for 30 minutes; verify the alert fires and the projector recovers without duplicate processing once resumed (idempotency key =event.id)."]
4.2 Drill checklist#
For every drill:
- Schedule announced ≥48h ahead in
#oncall-<domain>and on the status page. - Pre-drill snapshot of the dashboard captured.
- Drill IC and observer roles assigned (different humans).
- Drill runs with a timer; abort criteria stated upfront.
- Post-drill blameless retro within 5 business days; gaps land as backlog tickets in the next sprint.
4.3 Drill frequency#
[FILL IN: typically Q1/Q3 for vendor outage + DB failover, Q2/Q4 for quota +
subscriber-lag. Reference the calendar in <runbook-link>.]
5. Escalation contacts#
Three-tier escalation. Use the lowest tier first; only escalate when the tier above hasn't engaged within the window.
| Tier | Trigger | Who | How | SLA |
|---|---|---|---|---|
| 0 | Anomaly, no user impact | Domain on-call | PagerDuty schedule [FILL IN: schedule URL] | Best-effort |
| 1 | SLO burn-rate alert, ≤25% blast radius | Domain IC | PagerDuty escalation policy [FILL IN: policy URL] | 15m |
| 2 | SLO breach OR ≥25% blast radius OR vendor SLA breach | Domain lead + Platform IC + Comms | PagerDuty + Slack #incident-<domain> + status-page update |
5m |
| 3 | Cross-domain incident OR data-loss risk OR security event | VP Engineering + Security on-call + Legal | PagerDuty + executive bridge | Immediate |
Specific contacts. Always cross-check with the on-call rotation at [FILL IN: schedule URL] — names rotate.
| Role | Who (current) | Backup | Slack | PagerDuty |
|---|---|---|---|---|
| Domain on-call | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Domain IC | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Domain lead | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Platform IC | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Vendor-account contact | [FILL IN: e.g. OpenAI TAM email + business-hours phone] | [FILL IN] | n/a | n/a |
| Security on-call | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
| Comms / status page | [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
6. Vendor SLA tracking#
Every external vendor in §3's outbound list has a contractual SLA. Track each one against the subsystem's own SLOs so a vendor-side breach is detected and credit is claimed within the contractual window.
| Vendor | Tier | Uptime SLA | Latency SLA | Credit window | Credit % | Contract URL | Account contact |
|---|---|---|---|---|---|---|---|
| [FILL IN] | [FILL IN: Enterprise / Pro / Free] | [FILL IN: 99.9%] | [FILL IN: e.g. p95 < 2s] | [FILL IN: 30d] | [FILL IN: 10–30%] | [FILL IN: link to signed contract in Drive/Notion] | [FILL IN] |
Detection. Each vendor's availability is sampled by the @oshun/http-client
request metrics — see the architecture-overview
for the trace path. The Grafana panel at [FILL IN: panel link] surfaces the
per-vendor success rate over the credit window.
Claiming credit. When the per-vendor success rate falls below the SLA
threshold, file the support ticket within the credit window. The ticket template
lives at [FILL IN: e.g. docs/<domain>/runbooks/vendor-sla-claim-template.md].
Vendor-side rate-limit changes. Subscribe to the vendor's status page and changelog. Document the channels in the table below — when a vendor silently raises a rate limit, the soft ceiling in §3 must be updated in the same PR that lands the new behaviour.
7. Cost guardrails#
The §2.6 quality bar requires cost-aware operation. This section defines the tenant-level and platform-level guardrails for this subsystem.
7.1 Per-tenant quotas#
Per-tenant cost ceilings flow through the contract documented in
isis-cost-tracking-contract.md
(V1-P2-0005). For this subsystem:
| Tenant tier | Daily cap | Monthly cap | Behaviour at cap | Override path |
|---|---|---|---|---|
| Free | [FILL IN: e.g. $0.50] | [FILL IN: $5] | Reject with 402 + X-Quota-Reset header |
Self-service upgrade |
| Pro | [FILL IN] | [FILL IN] | Throttle to 50% rate, alert account | Account team |
| Enterprise | [FILL IN] | [FILL IN] | Soft alert at 80%, IC ack at 100% | Contract renegotiation |
Quota observability. Grafana panel [FILL IN: panel link] shows the top-N tenants by burn rate over the previous 24h. The isis/generation-pipeline.json dashboard's "cost-quota burn" panel is the reference shape.
7.2 Platform-level cost ceiling#
A daily platform-wide cap protects against quota-bypass bugs and runaway cron loops. The ceiling here is the maximum the subsystem can spend in 24h before paging.
| Period | Soft ceiling | Hard ceiling | Page-on-breach |
|---|---|---|---|
| Hour | [FILL IN: e.g. $50] | [FILL IN: e.g. $150] | Tier-1 |
| Day | [FILL IN: e.g. $500] | [FILL IN: e.g. $1500] | Tier-2 |
Hard-ceiling behaviour. [FILL IN: e.g. "All outbound provider calls are
short-circuited at the gateway with 503 quota_exhausted. Existing in-flight
requests are allowed to drain. Recovery is manual: ack the incident, raise the
ceiling, document the new limit here in the same PR that closes the incident."]
7.3 Scheduled cost reviews#
Every Tier-A subsystem is reviewed weekly for cost trend; monthly for quota cap accuracy.
- Weekly review: [FILL IN: meeting time, link to retro doc, dashboard URL].
- Monthly review: [FILL IN: meeting time, attendee list, agenda doc].
- Cost-anomaly alert: [FILL IN: PagerDuty service + threshold, e.g. "+30% WoW for any single tenant, page domain on-call within 5m"].
8. Deployment + rollback#
Reference [FILL IN: e.g. docs/<domain>/runbooks/deployment.md] for the deploy
procedure. The rollback path must be documented here even if the deploy
procedure lives elsewhere.
| Step | Action | Verify | Owner |
|---|---|---|---|
| 1 | [FILL IN: e.g. "Pause the BullMQ <queue> workers via pnpm nx run <subsystem>:scale --replicas=0"] |
[FILL IN] | On-call |
| 2 | [FILL IN: e.g. "Roll back deployment to the previous tagged release"] | [FILL IN] | On-call |
| 3 | [FILL IN: e.g. "Replay missed events from the Redis Stream up to the rollback timestamp"] | [FILL IN] | On-call |
| 4 | [FILL IN: e.g. "Resume workers; confirm SLO panels green"] | [FILL IN] | On-call |
9. Common alerts and first-response#
For each alert that pages this subsystem, document the runbook entry the alert links to. An alert without a runbook entry should not be on the on-call rotation.
| Alert | Severity | Runbook anchor | First-response |
|---|---|---|---|
[FILL IN: e.g. SLOAvailability1hBurn] |
Tier-1 | [FILL IN: link to a §10 entry] | [FILL IN: e.g. "Open oshun/platform-overview.json, check the per-vendor success-rate panel; if a single vendor is at 0%, run §4.1 scenario 1 (vendor outage)"] |
| [FILL IN] | [FILL IN] | [FILL IN] | [FILL IN] |
10. Diagnostic playbooks#
One subsection per recurring issue. Replace each ## 10.x [FILL IN] heading
with the symptom; below it document the steps to confirm the cause and the
action to take.
10.1 [FILL IN: symptom]#
Confirm cause. [FILL IN: queries, panel checks, log greps.]
Action. [FILL IN: explicit commands.]
Verify recovery. [FILL IN: SLO panel returning to green, queue depth draining, etc.]
10.2 [FILL IN: symptom]#
…
11. Verification — runbook completeness#
Before the runbook is marked Active, the domain owner must verify:
- Every
[FILL IN]placeholder above has been replaced with a concrete value (a string, a link, orn/awith a reason). - Every dashboard URL resolves and the linked panel exists.
- Every PagerDuty link resolves to a non-empty schedule or policy.
- Every vendor entry in §6 has a contract URL (drive/notion/etc.) that a reviewer can click.
- Every alert in §9 maps to a §10 playbook anchor.
- An outage drill (§4) has been executed at least once and the retro is linked.
- The §3 rate limits have been load-tested at 1.5× the soft ceiling and the dashboard screenshot is linked in the verification PR.
A runbook that fails any of these checks is Status: Draft and cannot be
referenced as an authoritative source during an incident.
12. Change log#
When updating the runbook:
- Increment
Datein the header. - Append an entry below.
- Land the change in a PR that another domain on-call has reviewed.
| Date | Author | Change |
|---|---|---|
<YYYY-MM-DD> |
<github handle> |
Initial fill from docs/releases/p2/runbook-template.md (V1-P2-0372). |
This template is the source of truth for runbook structure. When the template changes, every existing runbook is reviewed within 30 days for sections that need backfill. The change-log row in §12 is the audit trail.