Eve library

Runbook — `<SUBSYSTEM-NAME>`

Guide

Revision 1

On this page

Status: Draft | Active | Deprecated  Date: <YYYY-MM-DD>  Owner: eng-<domain>  Backlog item: <V1-P2-NNNN>  Audience: on-call SRE, incident commander, domain leads

This file is the V1-P2-0372 runbook template. Every operationally-owned subsystem in docs/releases/v1/specs/todos-p2.md Part III (Tier-A replacements) must own a populated copy at docs/<domain>/runbooks/<subsystem>.md. The header above is mandatory — if any of Status, Date, Owner, Backlog item, Audience is missing the runbook is not considered complete. Fill every [FILL IN] block; empty placeholders are not acceptable. When you fill the file in, replace this blockquote with a one-paragraph summary of the subsystem.

1. Subsystem at a glance#

  • Purpose. [FILL IN: one sentence describing what this subsystem does for the platform. e.g. "Issues short-lived OAuth tokens for vendor APIs used by Lilith and Yemaya."]
  • Code locations. [FILL IN: every libs/<x>/... and apps/<x>/... path that constitutes this subsystem. Be exhaustive — readers use this to grep.]
  • Owning team. [FILL IN: Slack #<channel>, GitHub team @<org>/<team>, rotating on-call schedule URL.]
  • Upstream consumers. [FILL IN: every domain or service that calls this subsystem — used to scope blast radius during incidents.]
  • Downstream dependencies. [FILL IN: every external API, internal service, or shared infrastructure component this subsystem talks to. See §3 (rate limits) and §6 (vendor SLA) for the contract details.]

2. SLOs and error budgets#

State the user-facing service-level objectives in absolute terms (latency percentile, success rate, freshness window) and link to the dashboard panel that displays them.

Metric SLO Window Burn-rate alert Dashboard panel
Availability [FILL IN: e.g. 99.5%] rolling 30d 14.4× over 1h [FILL IN: link to docker/observability/grafana/dashboards/<domain>/<subsystem>.json panel]
p95 latency [FILL IN: e.g. ≤ 750ms] rolling 7d 14.4× over 1h [FILL IN]
Error rate [FILL IN: e.g. < 0.5%] rolling 1h 6× over 5m [FILL IN]

Error budget policy. [FILL IN: e.g. "If the 30-day budget is consumed, no non-critical changes ship until budget recovers above 50%. Hot-fixes require IC sign-off."]

3. Rate limits#

Every external API and internal service has a rate limit. Document the hard ceiling, the soft ceiling, who hits it first, and how the subsystem behaves at the ceiling.

Direction Counterparty Hard limit Soft limit Bucket key Behaviour at limit
Outbound [FILL IN: e.g. OpenAI chat.completions] [FILL IN: e.g. 10k req/min, 90M tok/min, per org] [FILL IN: e.g. 8k req/min] [FILL IN: tenant or global] [FILL IN: e.g. "circuit breaker opens; failover-manager (libs/iris/failover) routes to next provider"]
Outbound [FILL IN] [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Inbound [FILL IN: e.g. external tenants] [FILL IN: e.g. 1000 req/min per tenant, enforced at API gateway] [FILL IN] [FILL IN] [FILL IN: e.g. "HTTP 429 with Retry-After; SLO error rate excludes 429s"]

Source of truth.

  • Outbound limits live in [FILL IN: e.g. libs/<domain>/<subsystem>/config.ts] and are enforced by @oshun/http-client retry + circuit-breaker config.
  • Inbound limits live in [FILL IN: e.g. the API gateway middleware at apps/<domain>/api/src/middleware/rate-limit.ts].
  • Vendor-side limits are tracked in §6 against the contractual SLA.

Verification. Load test at 1.5× the soft limit; confirm the circuit breaker opens and the failover path engages without leaking 5xx to the caller. Capture a screenshot of the dashboard during the test and link it in the rollout PR.

4. Outage drill#

A drill is a planned, scheduled exercise where the on-call team practises recovery from a realistic failure, with traffic in production or staging under controlled conditions. Drills are mandatory for every Tier-A subsystem, run at least once per quarter.

4.1 Drill scenarios#

The drill must cover every failure mode you'd page on. At minimum:

  1. Vendor outage. [FILL IN: "Block egress to <vendor> at the network layer for 10 minutes during off-peak. Verify failover-manager routes to the secondary provider and the SLO panel does not breach the burn-rate alert."]
  2. Database failover. [FILL IN: "Trigger a manual primary-replica failover on the <domain>_db PostgreSQL instance. Verify the BFF / API layer reconnects within 30s, that no data is lost, and that the subscriber-projection backlog clears within 5m."]
  3. Cache stampede. [FILL IN: e.g. "Flush the Redis cache for the subsystem and observe whether downstream amplification triggers circuit-breakers or an LLM cost spike."]
  4. Quota exhaustion. [FILL IN: e.g. "Force a single tenant to exceed the cost-tracking quota documented in docs/releases/p2/isis-cost-tracking-contract.md. Verify the quota middleware refuses the request before the provider call leaves the gateway."]
  5. Subscriber lag. [FILL IN: e.g. "Pause the <subsystem>-projector consumer for 30 minutes; verify the alert fires and the projector recovers without duplicate processing once resumed (idempotency key = event.id)."]

4.2 Drill checklist#

For every drill:

  • Schedule announced ≥48h ahead in #oncall-<domain> and on the status page.
  • Pre-drill snapshot of the dashboard captured.
  • Drill IC and observer roles assigned (different humans).
  • Drill runs with a timer; abort criteria stated upfront.
  • Post-drill blameless retro within 5 business days; gaps land as backlog tickets in the next sprint.

4.3 Drill frequency#

[FILL IN: typically Q1/Q3 for vendor outage + DB failover, Q2/Q4 for quota + subscriber-lag. Reference the calendar in <runbook-link>.]

5. Escalation contacts#

Three-tier escalation. Use the lowest tier first; only escalate when the tier above hasn't engaged within the window.

Tier Trigger Who How SLA
0 Anomaly, no user impact Domain on-call PagerDuty schedule [FILL IN: schedule URL] Best-effort
1 SLO burn-rate alert, ≤25% blast radius Domain IC PagerDuty escalation policy [FILL IN: policy URL] 15m
2 SLO breach OR ≥25% blast radius OR vendor SLA breach Domain lead + Platform IC + Comms PagerDuty + Slack #incident-<domain> + status-page update 5m
3 Cross-domain incident OR data-loss risk OR security event VP Engineering + Security on-call + Legal PagerDuty + executive bridge Immediate

Specific contacts. Always cross-check with the on-call rotation at [FILL IN: schedule URL] — names rotate.

Role Who (current) Backup Slack PagerDuty
Domain on-call [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Domain IC [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Domain lead [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Platform IC [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Vendor-account contact [FILL IN: e.g. OpenAI TAM email + business-hours phone] [FILL IN] n/a n/a
Security on-call [FILL IN] [FILL IN] [FILL IN] [FILL IN]
Comms / status page [FILL IN] [FILL IN] [FILL IN] [FILL IN]

6. Vendor SLA tracking#

Every external vendor in §3's outbound list has a contractual SLA. Track each one against the subsystem's own SLOs so a vendor-side breach is detected and credit is claimed within the contractual window.

Vendor Tier Uptime SLA Latency SLA Credit window Credit % Contract URL Account contact
[FILL IN] [FILL IN: Enterprise / Pro / Free] [FILL IN: 99.9%] [FILL IN: e.g. p95 < 2s] [FILL IN: 30d] [FILL IN: 10–30%] [FILL IN: link to signed contract in Drive/Notion] [FILL IN]

Detection. Each vendor's availability is sampled by the @oshun/http-client request metrics — see the architecture-overview for the trace path. The Grafana panel at [FILL IN: panel link] surfaces the per-vendor success rate over the credit window.

Claiming credit. When the per-vendor success rate falls below the SLA threshold, file the support ticket within the credit window. The ticket template lives at [FILL IN: e.g. docs/<domain>/runbooks/vendor-sla-claim-template.md].

Vendor-side rate-limit changes. Subscribe to the vendor's status page and changelog. Document the channels in the table below — when a vendor silently raises a rate limit, the soft ceiling in §3 must be updated in the same PR that lands the new behaviour.

7. Cost guardrails#

The §2.6 quality bar requires cost-aware operation. This section defines the tenant-level and platform-level guardrails for this subsystem.

7.1 Per-tenant quotas#

Per-tenant cost ceilings flow through the contract documented in isis-cost-tracking-contract.md (V1-P2-0005). For this subsystem:

Tenant tier Daily cap Monthly cap Behaviour at cap Override path
Free [FILL IN: e.g. $0.50] [FILL IN: $5] Reject with 402 + X-Quota-Reset header Self-service upgrade
Pro [FILL IN] [FILL IN] Throttle to 50% rate, alert account Account team
Enterprise [FILL IN] [FILL IN] Soft alert at 80%, IC ack at 100% Contract renegotiation

Quota observability. Grafana panel [FILL IN: panel link] shows the top-N tenants by burn rate over the previous 24h. The isis/generation-pipeline.json dashboard's "cost-quota burn" panel is the reference shape.

7.2 Platform-level cost ceiling#

A daily platform-wide cap protects against quota-bypass bugs and runaway cron loops. The ceiling here is the maximum the subsystem can spend in 24h before paging.

Period Soft ceiling Hard ceiling Page-on-breach
Hour [FILL IN: e.g. $50] [FILL IN: e.g. $150] Tier-1
Day [FILL IN: e.g. $500] [FILL IN: e.g. $1500] Tier-2

Hard-ceiling behaviour. [FILL IN: e.g. "All outbound provider calls are short-circuited at the gateway with 503 quota_exhausted. Existing in-flight requests are allowed to drain. Recovery is manual: ack the incident, raise the ceiling, document the new limit here in the same PR that closes the incident."]

7.3 Scheduled cost reviews#

Every Tier-A subsystem is reviewed weekly for cost trend; monthly for quota cap accuracy.

  • Weekly review: [FILL IN: meeting time, link to retro doc, dashboard URL].
  • Monthly review: [FILL IN: meeting time, attendee list, agenda doc].
  • Cost-anomaly alert: [FILL IN: PagerDuty service + threshold, e.g. "+30% WoW for any single tenant, page domain on-call within 5m"].

8. Deployment + rollback#

Reference [FILL IN: e.g. docs/<domain>/runbooks/deployment.md] for the deploy procedure. The rollback path must be documented here even if the deploy procedure lives elsewhere.

Step Action Verify Owner
1 [FILL IN: e.g. "Pause the BullMQ <queue> workers via pnpm nx run <subsystem>:scale --replicas=0"] [FILL IN] On-call
2 [FILL IN: e.g. "Roll back deployment to the previous tagged release"] [FILL IN] On-call
3 [FILL IN: e.g. "Replay missed events from the Redis Stream up to the rollback timestamp"] [FILL IN] On-call
4 [FILL IN: e.g. "Resume workers; confirm SLO panels green"] [FILL IN] On-call

9. Common alerts and first-response#

For each alert that pages this subsystem, document the runbook entry the alert links to. An alert without a runbook entry should not be on the on-call rotation.

Alert Severity Runbook anchor First-response
[FILL IN: e.g. SLOAvailability1hBurn] Tier-1 [FILL IN: link to a §10 entry] [FILL IN: e.g. "Open oshun/platform-overview.json, check the per-vendor success-rate panel; if a single vendor is at 0%, run §4.1 scenario 1 (vendor outage)"]
[FILL IN] [FILL IN] [FILL IN] [FILL IN]

10. Diagnostic playbooks#

One subsection per recurring issue. Replace each ## 10.x [FILL IN] heading with the symptom; below it document the steps to confirm the cause and the action to take.

10.1 [FILL IN: symptom]#

Confirm cause. [FILL IN: queries, panel checks, log greps.]

Action. [FILL IN: explicit commands.]

Verify recovery. [FILL IN: SLO panel returning to green, queue depth draining, etc.]

10.2 [FILL IN: symptom]#

…

11. Verification — runbook completeness#

Before the runbook is marked Active, the domain owner must verify:

  • Every [FILL IN] placeholder above has been replaced with a concrete value (a string, a link, or n/a with a reason).
  • Every dashboard URL resolves and the linked panel exists.
  • Every PagerDuty link resolves to a non-empty schedule or policy.
  • Every vendor entry in §6 has a contract URL (drive/notion/etc.) that a reviewer can click.
  • Every alert in §9 maps to a §10 playbook anchor.
  • An outage drill (§4) has been executed at least once and the retro is linked.
  • The §3 rate limits have been load-tested at 1.5× the soft ceiling and the dashboard screenshot is linked in the verification PR.

A runbook that fails any of these checks is Status: Draft and cannot be referenced as an authoritative source during an incident.

12. Change log#

When updating the runbook:

  1. Increment Date in the header.
  2. Append an entry below.
  3. Land the change in a PR that another domain on-call has reviewed.
Date Author Change
<YYYY-MM-DD> <github handle> Initial fill from docs/releases/p2/runbook-template.md (V1-P2-0372).

This template is the source of truth for runbook structure. When the template changes, every existing runbook is reviewed within 30 days for sections that need backfill. The change-log row in §12 is the audit trail.