Last Updated: 2026-02-22
Scope: Isis RunPod endpoint outages and RunPod cost spikes
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team
Purpose#
Provide a deterministic on-call response for:
- RunPod endpoint availability incidents (submit failures, queue backlog, execution failures, timeout storms).
- RunPod cost anomalies (unexpected per-second spend acceleration, runaway worker scale, canary misconfiguration).
This runbook is Isis-specific and assumes RunPod endpoint management is controlled by:
infra/runpod/endpoints/desired-state.jsonscripts/isis/runpod_endpoint_lifecycle.pyscripts/isis/detect_runpod_endpoint_drift.pyscripts/isis/reconcile_runpod_endpoint_drift.sh- generation API provider routing feature flags (
/api/v1/jobs/feature-flags)
Primary CloudWatch dashboards for this runbook:
oshun-<environment>-isis-provider-operations(provider latency and failures)oshun-<environment>-isis-cost(cost by endpoint, project, workflow, and user)oshun-<environment>-runpod(endpoint queue, utilization, and cost)
Observability stack contract validation command (IaC wiring check):
node scripts/isis/validate_observability_stack.mjs
Distributed trace stage events emitted by Isis services:
trace.queue.enqueuetrace.job.lifecycletrace.worker.executetrace.worker.completedtrace.worker.failedtrace.provider.dispatchtrace.provider.completedtrace.provider.failed
Alert Triggers#
Declare incident when one or more of the following are true:
- CloudWatch alarm
oshun-<environment>-isis-provider-cost-spike-anomalyentersALARM. - CloudWatch alarm
oshun-<environment>-isis-provider-failure-rate-spike-anomalyentersALARM. - RunPod-related failure rate > 10% for 10 minutes.
- Submit-to-start latency p95 > 2x baseline for 10 minutes.
- Queue growth is monotonic for 15 minutes while completion throughput is flat.
- RunPod cost burn > 20% above forecast over 30 minutes.
- Endpoint drift detected on production and not resolved after reconcile.
- Canary endpoint enabled with unresolved endpoint ID or missing endpoint.
Severity and Ownership#
| Severity | Examples | Owner |
|---|---|---|
| SEV-1 | All RunPod endpoints unavailable; generation outage | @oshun/devops incident lead + @oshun/isis-team |
| SEV-2 | Partial endpoint class outage; sustained timeout spike | @oshun/devops |
| SEV-2 | Cost burn spike with no throughput gain | @oshun/devops + @oshun/ml-team |
| SEV-3 | Canary misrouting or limited endpoint drift | @oshun/devops |
15-Minute Triage Checklist#
- Confirm blast radius from Isis API:
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/providers/operations" | jq
Operator CLI equivalent (requires queue-admin scope):
isis jobs triage --max-pages 10 --dead-letter-limit 25
- Check RunPod endpoint drift:
python3 scripts/isis/detect_runpod_endpoint_drift.py \
--desired-state-file infra/runpod/endpoints/desired-state.json \
--environment production \
--report-file /tmp/runpod-drift-report.json
jq . /tmp/runpod-drift-report.json
- Check current canary configuration and percentages:
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags" | jq '.routing.endpointCanary'
- Run canary SLO/cost verification report:
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh
- If evaluating canary promotion, generate promote/rollback decision:
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
- Check active budget guardrail mode:
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod.guardrail'
- Determine if the issue is:
- endpoint outage/capacity failure
- misconfiguration/drift
- canary regression
- cost-only anomaly with healthy throughput
- If dead-letter entries are present and root cause is mitigated, replay:
isis jobs replay --all --reason "post-incident recovery replay"
- Start incident comms and assign incident lead + scribe.
RunPod Endpoint Outage Playbook#
A. Immediate Containment#
- Stop canary exposure (first mitigation):
- Non-production: set canary percentages to
0through feature flag override. - Production: set
RUNPOD_*_CANARY_PERCENTAGE=0in deployment config and roll generation API.
- Non-production: set canary percentages to
- If primary RunPod path is unhealthy, fail over provider routing:
- set
COMFYUI_PROVIDER=runcomfy, or - disable migration with
ENABLE_RUNPOD_MIGRATION=false.
- set
- Freeze endpoint lifecycle changes until baseline stabilizes.
B. Infrastructure Reconciliation#
- Reconcile endpoint drift:
ENVIRONMENT=production DRY_RUN=false ALLOW_MISSING_ENDPOINT_ID=false \
bash scripts/isis/reconcile_runpod_endpoint_drift.sh
- If reconcile fails, apply desired state directly:
python3 scripts/isis/runpod_endpoint_lifecycle.py \
--operation apply-desired-state \
--desired-state-file infra/runpod/endpoints/desired-state.json \
--environment production
- Verify endpoint recovery:
python3 scripts/isis/detect_runpod_endpoint_drift.py \
--desired-state-file infra/runpod/endpoints/desired-state.json \
--environment production \
--report-file /tmp/runpod-drift-post.json
jq '.driftCount, .drift' /tmp/runpod-drift-post.json
C. Recovery Validation#
- Run smoke generations on primary workflows.
- Validate terminal status distribution returns to baseline.
- Validate queue depth decreases and active jobs complete.
- Keep canary at
0%until postmortem and fix validation are complete.
RunPod Cost Spike Playbook#
A. Confirm the Spike#
- Validate burn trend and throughput:
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod | {usage, costs, endpointCanary}'
- Validate no equivalent throughput gain in completed jobs.
- Correlate spike with recent changes:
- endpoint lifecycle updates
- canary percentage increases
- worker max/idle timeout changes
B. Immediate Cost Controls#
- Set all canary percentages to
0(reduce change risk and double-run burn). - Reduce workers on expensive endpoint classes in desired state:
- lower
workersMax - lower
workersMinwhere safe
- lower
- Re-apply desired state after edits.
- If required, force provider fallback off RunPod until costs stabilize.
C. Root-Cause Patterns#
| Pattern | Signal | Remediation |
|---|---|---|
| Canary misconfiguration | Canary endpoint active with high percentage and regression | set canary % to 0, fix endpoint image/config, re-stage |
| Worker overprovisioning | high idle spend, low throughput | lower workersMin, lower idleTimeout, lower workersMax |
| Retry storm | high retries and failure loops | cap retries, fail over provider, isolate failing endpoint |
| Drifted timeout settings | runaway execution windows | restore desired execution/idle timeout via reconcile |
Feature Flag Emergency Commands (Non-Prod Overrides)#
# Disable ComfyUI canary traffic immediately
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/RUNPOD_COMFYUI_CANARY_PERCENTAGE" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
-d '{"value":0}'
# Disable migration to RunPod
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/ENABLE_RUNPOD_MIGRATION" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
-d '{"value":false}'
Note: production feature-flag overrides are read-only by design. Use deployment configuration/environment variable updates in production.
Communication Templates#
Incident Start#
RunPod Incident started (SEV-X). Scope: <affected endpoints/workflows>. Mitigation in progress: canary=0%, drift reconcile started. Next update in 15m.
Incident Update#
RunPod Incident update: <metrics delta>. Actions completed: <A,B,C>. Remaining risk: <risk>. ETA next checkpoint: <time>.
Incident Resolved#
RunPod Incident resolved. Recovery validated on queue, failure rate, latency, and cost trend. Canary remains at 0% pending postmortem actions.
Exit Criteria#
Incident can close only when all are true:
- Failure rate and latency are back within normal guardrails for 30 minutes.
- Queue backlog is draining and not growing.
- Drift report is clean or accepted exceptions are documented.
- Cost burn has normalized or active budget controls are in place.
- Follow-up remediation tasks are filed and prioritized.
Post-Incident Review Requirement#
For SEV-1/SEV-2 incidents and any confirmed cost overrun event, complete a
post-incident review using:
docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md
Related Documentation#
docs/infrastructure/runpod-integration.mddocs/domains/isis/runbooks/runpod-credential-rotation.mddocs/domains/isis/runbooks/provider-outage-drill.mddocs/domains/isis/runbooks/endpoint-region-failover-drill.mddocs/domains/isis/runbooks/runpod-dr-gameday.mddocs/domains/isis/runbooks/runpod-budget-guardrails.mddocs/domains/isis/runbooks/creative-pipeline-oncall.mddocs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.mddocs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.mddocs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.mddocs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.mddocs/reference/runbooks/incident-response.mddocs/reference/runbooks/cost-management.md