# Isis RunPod On-Call Runbook

Last Updated: 2026-02-22  
Scope: Isis RunPod endpoint outages and RunPod cost spikes  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`, `@oshun/ml-team`

## Purpose

Provide a deterministic on-call response for:

1. RunPod endpoint availability incidents (submit failures, queue backlog,
   execution failures, timeout storms).
2. RunPod cost anomalies (unexpected per-second spend acceleration, runaway
   worker scale, canary misconfiguration).

This runbook is Isis-specific and assumes RunPod endpoint management is
controlled by:

- `infra/runpod/endpoints/desired-state.json`
- `scripts/isis/runpod_endpoint_lifecycle.py`
- `scripts/isis/detect_runpod_endpoint_drift.py`
- `scripts/isis/reconcile_runpod_endpoint_drift.sh`
- generation API provider routing feature flags (`/api/v1/jobs/feature-flags`)

Primary CloudWatch dashboards for this runbook:

- `oshun-<environment>-isis-provider-operations` (provider latency and failures)
- `oshun-<environment>-isis-cost` (cost by endpoint, project, workflow, and
  user)
- `oshun-<environment>-runpod` (endpoint queue, utilization, and cost)

Observability stack contract validation command (IaC wiring check):

```bash
node scripts/isis/validate_observability_stack.mjs
```

Distributed trace stage events emitted by Isis services:

- `trace.queue.enqueue`
- `trace.job.lifecycle`
- `trace.worker.execute`
- `trace.worker.completed`
- `trace.worker.failed`
- `trace.provider.dispatch`
- `trace.provider.completed`
- `trace.provider.failed`

## Alert Triggers

Declare incident when one or more of the following are true:

- CloudWatch alarm `oshun-<environment>-isis-provider-cost-spike-anomaly` enters
  `ALARM`.
- CloudWatch alarm
  `oshun-<environment>-isis-provider-failure-rate-spike-anomaly` enters `ALARM`.
- RunPod-related failure rate > 10% for 10 minutes.
- Submit-to-start latency p95 > 2x baseline for 10 minutes.
- Queue growth is monotonic for 15 minutes while completion throughput is flat.
- RunPod cost burn > 20% above forecast over 30 minutes.
- Endpoint drift detected on production and not resolved after reconcile.
- Canary endpoint enabled with unresolved endpoint ID or missing endpoint.

## Severity and Ownership

| Severity | Examples                                               | Owner                                              |
| -------- | ------------------------------------------------------ | -------------------------------------------------- |
| SEV-1    | All RunPod endpoints unavailable; generation outage    | `@oshun/devops` incident lead + `@oshun/isis-team` |
| SEV-2    | Partial endpoint class outage; sustained timeout spike | `@oshun/devops`                                    |
| SEV-2    | Cost burn spike with no throughput gain                | `@oshun/devops` + `@oshun/ml-team`                 |
| SEV-3    | Canary misrouting or limited endpoint drift            | `@oshun/devops`                                    |

## 15-Minute Triage Checklist

1. Confirm blast radius from Isis API:

```bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/providers/operations" | jq
```

Operator CLI equivalent (requires queue-admin scope):

```bash
isis jobs triage --max-pages 10 --dead-letter-limit 25
```

2. Check RunPod endpoint drift:

```bash
python3 scripts/isis/detect_runpod_endpoint_drift.py \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production \
  --report-file /tmp/runpod-drift-report.json
jq . /tmp/runpod-drift-report.json
```

3. Check current canary configuration and percentages:

```bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags" | jq '.routing.endpointCanary'
```

4. Run canary SLO/cost verification report:

```bash
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh
```

5. If evaluating canary promotion, generate promote/rollback decision:

```bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
```

6. Check active budget guardrail mode:

```bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod.guardrail'
```

7. Determine if the issue is:
   - endpoint outage/capacity failure
   - misconfiguration/drift
   - canary regression
   - cost-only anomaly with healthy throughput
8. If dead-letter entries are present and root cause is mitigated, replay:

```bash
isis jobs replay --all --reason "post-incident recovery replay"
```

9. Start incident comms and assign incident lead + scribe.

## RunPod Endpoint Outage Playbook

### A. Immediate Containment

1. Stop canary exposure (first mitigation):
   - Non-production: set canary percentages to `0` through feature flag
     override.
   - Production: set `RUNPOD_*_CANARY_PERCENTAGE=0` in deployment config and
     roll generation API.
2. If primary RunPod path is unhealthy, fail over provider routing:
   - set `COMFYUI_PROVIDER=runcomfy`, or
   - disable migration with `ENABLE_RUNPOD_MIGRATION=false`.
3. Freeze endpoint lifecycle changes until baseline stabilizes.

### B. Infrastructure Reconciliation

1. Reconcile endpoint drift:

```bash
ENVIRONMENT=production DRY_RUN=false ALLOW_MISSING_ENDPOINT_ID=false \
  bash scripts/isis/reconcile_runpod_endpoint_drift.sh
```

2. If reconcile fails, apply desired state directly:

```bash
python3 scripts/isis/runpod_endpoint_lifecycle.py \
  --operation apply-desired-state \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production
```

3. Verify endpoint recovery:

```bash
python3 scripts/isis/detect_runpod_endpoint_drift.py \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production \
  --report-file /tmp/runpod-drift-post.json
jq '.driftCount, .drift' /tmp/runpod-drift-post.json
```

### C. Recovery Validation

- Run smoke generations on primary workflows.
- Validate terminal status distribution returns to baseline.
- Validate queue depth decreases and active jobs complete.
- Keep canary at `0%` until postmortem and fix validation are complete.

## RunPod Cost Spike Playbook

### A. Confirm the Spike

1. Validate burn trend and throughput:

```bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod | {usage, costs, endpointCanary}'
```

2. Validate no equivalent throughput gain in completed jobs.
3. Correlate spike with recent changes:
   - endpoint lifecycle updates
   - canary percentage increases
   - worker max/idle timeout changes

### B. Immediate Cost Controls

1. Set all canary percentages to `0` (reduce change risk and double-run burn).
2. Reduce workers on expensive endpoint classes in desired state:
   - lower `workersMax`
   - lower `workersMin` where safe
3. Re-apply desired state after edits.
4. If required, force provider fallback off RunPod until costs stabilize.

### C. Root-Cause Patterns

| Pattern                  | Signal                                                     | Remediation                                                 |
| ------------------------ | ---------------------------------------------------------- | ----------------------------------------------------------- |
| Canary misconfiguration  | Canary endpoint active with high percentage and regression | set canary `%` to `0`, fix endpoint image/config, re-stage  |
| Worker overprovisioning  | high idle spend, low throughput                            | lower `workersMin`, lower `idleTimeout`, lower `workersMax` |
| Retry storm              | high retries and failure loops                             | cap retries, fail over provider, isolate failing endpoint   |
| Drifted timeout settings | runaway execution windows                                  | restore desired execution/idle timeout via reconcile        |

## Feature Flag Emergency Commands (Non-Prod Overrides)

```bash
# Disable ComfyUI canary traffic immediately
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/RUNPOD_COMFYUI_CANARY_PERCENTAGE" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
  -d '{"value":0}'

# Disable migration to RunPod
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/ENABLE_RUNPOD_MIGRATION" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
  -d '{"value":false}'
```

Note: production feature-flag overrides are read-only by design. Use deployment
configuration/environment variable updates in production.

## Communication Templates

### Incident Start

`RunPod Incident started (SEV-X). Scope: <affected endpoints/workflows>. Mitigation in progress: canary=0%, drift reconcile started. Next update in 15m.`

### Incident Update

`RunPod Incident update: <metrics delta>. Actions completed: <A,B,C>. Remaining risk: <risk>. ETA next checkpoint: <time>.`

### Incident Resolved

`RunPod Incident resolved. Recovery validated on queue, failure rate, latency, and cost trend. Canary remains at 0% pending postmortem actions.`

## Exit Criteria

Incident can close only when all are true:

- Failure rate and latency are back within normal guardrails for 30 minutes.
- Queue backlog is draining and not growing.
- Drift report is clean or accepted exceptions are documented.
- Cost burn has normalized or active budget controls are in place.
- Follow-up remediation tasks are filed and prioritized.

## Post-Incident Review Requirement

For SEV-1/SEV-2 incidents and any confirmed cost overrun event, complete a
post-incident review using:
`docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md`

## Related Documentation

- `docs/infrastructure/runpod-integration.md`
- `docs/domains/isis/runbooks/runpod-credential-rotation.md`
- `docs/domains/isis/runbooks/provider-outage-drill.md`
- `docs/domains/isis/runbooks/endpoint-region-failover-drill.md`
- `docs/domains/isis/runbooks/runpod-dr-gameday.md`
- `docs/domains/isis/runbooks/runpod-budget-guardrails.md`
- `docs/domains/isis/runbooks/creative-pipeline-oncall.md`
- `docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md`
- `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`
- `docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md`
- `docs/reference/runbooks/incident-response.md`
- `docs/reference/runbooks/cost-management.md`
