Eve library

Isis Provider Outage Disaster Recovery Drill Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-22
Scope: RunPod outage simulation with RunComfy fallback verification
Primary Owners: @oshun/devops, @oshun/isis-team

Purpose#

Run a repeatable disaster-recovery drill for RunPod provider outage scenarios and verify that Isis can fallback to non-RunPod provider routing without breaking generation flows.

Drill Objectives#

  1. Simulate RunPod outage conditions via routing controls.
  2. Verify fallback routing (runcomfy) is active and healthy.
  3. Verify synthetic generation submissions still succeed.
  4. Restore original routing state after drill completion.

Preconditions#

  • Execute in non-production environment where provider feature flags are writable.
  • ISIS_API_BASE_URL and privileged ISIS_ADMIN_TOKEN available.
  • RunComfy fallback provider is configured and healthy.
  • Change ticket opened with drill window and rollback owner.

Drill Procedure#

A. Dry-Run Validation (No Mutations)#

bash
DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh

Dry-run captures baseline snapshots only and marks mutation-dependent checks as skip.

B. Execute Outage Drill#

bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh

C. What the Drill Automation Does#

scripts/isis/run_provider_outage_drill.sh performs:

  1. Capture baseline /api/v1/jobs/feature-flags.
  2. Capture baseline /api/v1/jobs/providers/operations.
  3. Apply outage simulation flags:
    • COMFYUI_PROVIDER=runcomfy
    • ENABLE_RUNPOD_MIGRATION=false
    • RUNPOD_MIGRATION_PERCENTAGE=0
  4. Verify routing fallback flags.
  5. Verify /api/v1/jobs/providers/operations shows fallback provider health.
  6. Submit and poll synthetic drill job (optional).
  7. Restore baseline flags (if RESTORE_FLAGS=true).
  8. Verify restored flags match baseline.
  9. Emit JSON drill report in tmp/isis-provider-drill/.

D. Optional Environment Controls#

  • API_TIMEOUT_SECONDS (default 30) controls HTTP timeout for drill calls.
  • SUBMIT_SYNTHETIC_JOB=false skips synthetic submit/progress checks.
  • RESTORE_FLAGS=false disables restore for controlled experiments (not recommended for routine drills).
  • STRICT=true makes the script exit non-zero if any check fails.

Success Criteria#

Drill passes only if:

  • fallback flags are applied as expected during simulation.
  • provider operations indicate healthy fallback path.
  • synthetic job is accepted and reaches expected non-failed state.
  • original routing flags are restored successfully.
  • restored routing flags are verified equal to pre-drill baseline.

Failure Handling#

If drill fails:

  1. Immediately restore baseline feature flags.
  2. Validate provider operations and runpod operations snapshots.
  3. Open incident if fallback path is degraded.
  4. File remediation tasks before next drill.

Evidence Artifacts#

  • tmp/isis-provider-drill/provider-outage-drill-*.json
  • baseline/post-drill feature flag snapshots
  • baseline/post-drill provider operations snapshots
  • restored feature-flag verification snapshot
  • synthetic submit/status snapshots
  • Minimum: monthly in staging.
  • Additional runs:
    • before major provider routing changes
    • after RunPod migration/canary logic changes
    • after major incident involving provider instability.
  • scripts/isis/run_provider_outage_drill.sh
  • scripts/isis/run_runpod_disaster_recovery_gameday.sh
  • docs/domains/isis/runbooks/runpod-oncall.md
  • docs/domains/isis/runbooks/endpoint-region-failover-drill.md
  • docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md