Last Updated: 2026-02-22
Scope: RunPod outage simulation with RunComfy fallback verification
Primary Owners: @oshun/devops, @oshun/isis-team
Purpose#
Run a repeatable disaster-recovery drill for RunPod provider outage scenarios and verify that Isis can fallback to non-RunPod provider routing without breaking generation flows.
Drill Objectives#
- Simulate RunPod outage conditions via routing controls.
- Verify fallback routing (
runcomfy) is active and healthy. - Verify synthetic generation submissions still succeed.
- Restore original routing state after drill completion.
Preconditions#
- Execute in non-production environment where provider feature flags are writable.
ISIS_API_BASE_URLand privilegedISIS_ADMIN_TOKENavailable.- RunComfy fallback provider is configured and healthy.
- Change ticket opened with drill window and rollback owner.
Drill Procedure#
A. Dry-Run Validation (No Mutations)#
bash
DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh
Dry-run captures baseline snapshots only and marks mutation-dependent checks as
skip.
B. Execute Outage Drill#
bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh
C. What the Drill Automation Does#
scripts/isis/run_provider_outage_drill.sh performs:
- Capture baseline
/api/v1/jobs/feature-flags. - Capture baseline
/api/v1/jobs/providers/operations. - Apply outage simulation flags:
COMFYUI_PROVIDER=runcomfyENABLE_RUNPOD_MIGRATION=falseRUNPOD_MIGRATION_PERCENTAGE=0
- Verify routing fallback flags.
- Verify
/api/v1/jobs/providers/operationsshows fallback provider health. - Submit and poll synthetic drill job (optional).
- Restore baseline flags (if
RESTORE_FLAGS=true). - Verify restored flags match baseline.
- Emit JSON drill report in
tmp/isis-provider-drill/.
D. Optional Environment Controls#
API_TIMEOUT_SECONDS(default30) controls HTTP timeout for drill calls.SUBMIT_SYNTHETIC_JOB=falseskips synthetic submit/progress checks.RESTORE_FLAGS=falsedisables restore for controlled experiments (not recommended for routine drills).STRICT=truemakes the script exit non-zero if any check fails.
Success Criteria#
Drill passes only if:
- fallback flags are applied as expected during simulation.
- provider operations indicate healthy fallback path.
- synthetic job is accepted and reaches expected non-failed state.
- original routing flags are restored successfully.
- restored routing flags are verified equal to pre-drill baseline.
Failure Handling#
If drill fails:
- Immediately restore baseline feature flags.
- Validate provider operations and runpod operations snapshots.
- Open incident if fallback path is degraded.
- File remediation tasks before next drill.
Evidence Artifacts#
tmp/isis-provider-drill/provider-outage-drill-*.json- baseline/post-drill feature flag snapshots
- baseline/post-drill provider operations snapshots
- restored feature-flag verification snapshot
- synthetic submit/status snapshots
Recommended Cadence#
- Minimum: monthly in staging.
- Additional runs:
- before major provider routing changes
- after RunPod migration/canary logic changes
- after major incident involving provider instability.
Related Artifacts#
scripts/isis/run_provider_outage_drill.shscripts/isis/run_runpod_disaster_recovery_gameday.shdocs/domains/isis/runbooks/runpod-oncall.mddocs/domains/isis/runbooks/endpoint-region-failover-drill.mddocs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md