Eve library

Isis RunPod Disaster Recovery Game-Day Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: Composite game-day for provider-failure and endpoint-region-failure scenarios
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team

Purpose#

Execute a deterministic game-day that validates Isis RunPod disaster-recovery readiness for:

  1. RunPod provider-level failure (fallback to RunComfy).
  2. Endpoint-family regional failure (canary-to-stable failover routing).

Scenario Matrix#

Scenario ID Failure Class Automation Script Outcome Gate
DR-GD-001 Provider failure scripts/isis/run_provider_outage_drill.sh overall.pass=true
DR-GD-002 Endpoint regional failover scripts/isis/run_endpoint_region_failover_drill.sh overall.pass=true

Preconditions#

  • Non-production environment with writable provider feature flags.
  • ISIS_API_BASE_URL and privileged ISIS_ADMIN_TOKEN.
  • RunComfy fallback provider configured and healthy.
  • Target endpoint families have canary endpoints provisioned.
  • Incident commander and rollback owner assigned.

Composite Execution#

Dry run#

bash
DRY_RUN=true \
STRICT=true \
REGION_FAMILIES="comfyui,flux" \
scripts/isis/run_runpod_disaster_recovery_gameday.sh

Live game-day#

bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_REGION_FAILURE_SCENARIO=true \
REGION_FAMILIES="comfyui,flux" \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_runpod_disaster_recovery_gameday.sh

Key Controls#

  • RESTORE_FLAGS=true: mandatory for routine exercises.
  • SUBMIT_SYNTHETIC_JOB=true: validates runtime path under injected failures.
  • REGION_FAMILIES: comma-separated endpoint families for regional failover drills.
  • STRICT=true: exits non-zero if any scenario fails.

Success Criteria#

Game-day passes only when:

  • each enabled scenario emits a valid report,
  • no scenario reports failed checks,
  • routing flags are restored to baseline,
  • synthetic validation succeeds or is explicitly skipped with approval.

Evidence Artifacts#

  • Composite report:
    • tmp/isis-runpod-dr-gameday/runpod-dr-gameday-*.json
  • Scenario reports:
    • tmp/isis-runpod-dr-gameday/provider-outage-drill.json
    • tmp/isis-runpod-dr-gameday/region-failover-<family>.json

Cadence#

  • Monthly in staging.
  • Mandatory before major endpoint topology or provider-routing changes.
  • Mandatory after SEV-1/SEV-2 incidents in provider routing.
  • scripts/isis/run_runpod_disaster_recovery_gameday.sh
  • scripts/isis/run_provider_outage_drill.sh
  • scripts/isis/run_endpoint_region_failover_drill.sh
  • docs/domains/isis/runbooks/provider-outage-drill.md
  • docs/domains/isis/runbooks/endpoint-region-failover-drill.md
  • docs/domains/isis/runbooks/runpod-oncall.md