Eve library

Isis RunPod On-Call Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-22
Scope: Isis RunPod endpoint outages and RunPod cost spikes
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team

Purpose#

Provide a deterministic on-call response for:

  1. RunPod endpoint availability incidents (submit failures, queue backlog, execution failures, timeout storms).
  2. RunPod cost anomalies (unexpected per-second spend acceleration, runaway worker scale, canary misconfiguration).

This runbook is Isis-specific and assumes RunPod endpoint management is controlled by:

  • infra/runpod/endpoints/desired-state.json
  • scripts/isis/runpod_endpoint_lifecycle.py
  • scripts/isis/detect_runpod_endpoint_drift.py
  • scripts/isis/reconcile_runpod_endpoint_drift.sh
  • generation API provider routing feature flags (/api/v1/jobs/feature-flags)

Primary CloudWatch dashboards for this runbook:

  • oshun-<environment>-isis-provider-operations (provider latency and failures)
  • oshun-<environment>-isis-cost (cost by endpoint, project, workflow, and user)
  • oshun-<environment>-runpod (endpoint queue, utilization, and cost)

Observability stack contract validation command (IaC wiring check):

bash
node scripts/isis/validate_observability_stack.mjs

Distributed trace stage events emitted by Isis services:

  • trace.queue.enqueue
  • trace.job.lifecycle
  • trace.worker.execute
  • trace.worker.completed
  • trace.worker.failed
  • trace.provider.dispatch
  • trace.provider.completed
  • trace.provider.failed

Alert Triggers#

Declare incident when one or more of the following are true:

  • CloudWatch alarm oshun-<environment>-isis-provider-cost-spike-anomaly enters ALARM.
  • CloudWatch alarm oshun-<environment>-isis-provider-failure-rate-spike-anomaly enters ALARM.
  • RunPod-related failure rate > 10% for 10 minutes.
  • Submit-to-start latency p95 > 2x baseline for 10 minutes.
  • Queue growth is monotonic for 15 minutes while completion throughput is flat.
  • RunPod cost burn > 20% above forecast over 30 minutes.
  • Endpoint drift detected on production and not resolved after reconcile.
  • Canary endpoint enabled with unresolved endpoint ID or missing endpoint.

Severity and Ownership#

Severity Examples Owner
SEV-1 All RunPod endpoints unavailable; generation outage @oshun/devops incident lead + @oshun/isis-team
SEV-2 Partial endpoint class outage; sustained timeout spike @oshun/devops
SEV-2 Cost burn spike with no throughput gain @oshun/devops + @oshun/ml-team
SEV-3 Canary misrouting or limited endpoint drift @oshun/devops

15-Minute Triage Checklist#

  1. Confirm blast radius from Isis API:
bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/providers/operations" | jq

Operator CLI equivalent (requires queue-admin scope):

bash
isis jobs triage --max-pages 10 --dead-letter-limit 25
  1. Check RunPod endpoint drift:
bash
python3 scripts/isis/detect_runpod_endpoint_drift.py \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production \
  --report-file /tmp/runpod-drift-report.json
jq . /tmp/runpod-drift-report.json
  1. Check current canary configuration and percentages:
bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags" | jq '.routing.endpointCanary'
  1. Run canary SLO/cost verification report:
bash
ISIS_DATABASE_URL=postgres://... \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
scripts/isis/verify_canary_slo_cost.sh
  1. If evaluating canary promotion, generate promote/rollback decision:
bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
  1. Check active budget guardrail mode:
bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod.guardrail'
  1. Determine if the issue is:
    • endpoint outage/capacity failure
    • misconfiguration/drift
    • canary regression
    • cost-only anomaly with healthy throughput
  2. If dead-letter entries are present and root cause is mitigated, replay:
bash
isis jobs replay --all --reason "post-incident recovery replay"
  1. Start incident comms and assign incident lead + scribe.

RunPod Endpoint Outage Playbook#

A. Immediate Containment#

  1. Stop canary exposure (first mitigation):
    • Non-production: set canary percentages to 0 through feature flag override.
    • Production: set RUNPOD_*_CANARY_PERCENTAGE=0 in deployment config and roll generation API.
  2. If primary RunPod path is unhealthy, fail over provider routing:
    • set COMFYUI_PROVIDER=runcomfy, or
    • disable migration with ENABLE_RUNPOD_MIGRATION=false.
  3. Freeze endpoint lifecycle changes until baseline stabilizes.

B. Infrastructure Reconciliation#

  1. Reconcile endpoint drift:
bash
ENVIRONMENT=production DRY_RUN=false ALLOW_MISSING_ENDPOINT_ID=false \
  bash scripts/isis/reconcile_runpod_endpoint_drift.sh
  1. If reconcile fails, apply desired state directly:
bash
python3 scripts/isis/runpod_endpoint_lifecycle.py \
  --operation apply-desired-state \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production
  1. Verify endpoint recovery:
bash
python3 scripts/isis/detect_runpod_endpoint_drift.py \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production \
  --report-file /tmp/runpod-drift-post.json
jq '.driftCount, .drift' /tmp/runpod-drift-post.json

C. Recovery Validation#

  • Run smoke generations on primary workflows.
  • Validate terminal status distribution returns to baseline.
  • Validate queue depth decreases and active jobs complete.
  • Keep canary at 0% until postmortem and fix validation are complete.

RunPod Cost Spike Playbook#

A. Confirm the Spike#

  1. Validate burn trend and throughput:
bash
curl -s "$ISIS_API_BASE_URL/api/v1/jobs/runpod/operations" | jq '.runpod | {usage, costs, endpointCanary}'
  1. Validate no equivalent throughput gain in completed jobs.
  2. Correlate spike with recent changes:
    • endpoint lifecycle updates
    • canary percentage increases
    • worker max/idle timeout changes

B. Immediate Cost Controls#

  1. Set all canary percentages to 0 (reduce change risk and double-run burn).
  2. Reduce workers on expensive endpoint classes in desired state:
    • lower workersMax
    • lower workersMin where safe
  3. Re-apply desired state after edits.
  4. If required, force provider fallback off RunPod until costs stabilize.

C. Root-Cause Patterns#

Pattern Signal Remediation
Canary misconfiguration Canary endpoint active with high percentage and regression set canary % to 0, fix endpoint image/config, re-stage
Worker overprovisioning high idle spend, low throughput lower workersMin, lower idleTimeout, lower workersMax
Retry storm high retries and failure loops cap retries, fail over provider, isolate failing endpoint
Drifted timeout settings runaway execution windows restore desired execution/idle timeout via reconcile

Feature Flag Emergency Commands (Non-Prod Overrides)#

bash
# Disable ComfyUI canary traffic immediately
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/RUNPOD_COMFYUI_CANARY_PERCENTAGE" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
  -d '{"value":0}'

# Disable migration to RunPod
curl -sS -X PATCH "$ISIS_API_BASE_URL/api/v1/jobs/feature-flags/ENABLE_RUNPOD_MIGRATION" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ISIS_ADMIN_TOKEN" \
  -d '{"value":false}'

Note: production feature-flag overrides are read-only by design. Use deployment configuration/environment variable updates in production.

Communication Templates#

Incident Start#

RunPod Incident started (SEV-X). Scope: <affected endpoints/workflows>. Mitigation in progress: canary=0%, drift reconcile started. Next update in 15m.

Incident Update#

RunPod Incident update: <metrics delta>. Actions completed: <A,B,C>. Remaining risk: <risk>. ETA next checkpoint: <time>.

Incident Resolved#

RunPod Incident resolved. Recovery validated on queue, failure rate, latency, and cost trend. Canary remains at 0% pending postmortem actions.

Exit Criteria#

Incident can close only when all are true:

  • Failure rate and latency are back within normal guardrails for 30 minutes.
  • Queue backlog is draining and not growing.
  • Drift report is clean or accepted exceptions are documented.
  • Cost burn has normalized or active budget controls are in place.
  • Follow-up remediation tasks are filed and prioritized.

Post-Incident Review Requirement#

For SEV-1/SEV-2 incidents and any confirmed cost overrun event, complete a post-incident review using: docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md

  • docs/infrastructure/runpod-integration.md
  • docs/domains/isis/runbooks/runpod-credential-rotation.md
  • docs/domains/isis/runbooks/provider-outage-drill.md
  • docs/domains/isis/runbooks/endpoint-region-failover-drill.md
  • docs/domains/isis/runbooks/runpod-dr-gameday.md
  • docs/domains/isis/runbooks/runpod-budget-guardrails.md
  • docs/domains/isis/runbooks/creative-pipeline-oncall.md
  • docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md
  • docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
  • docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md
  • docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md
  • docs/reference/runbooks/incident-response.md
  • docs/reference/runbooks/cost-management.md