Eve library

ISIS v2 WS09-010: RunPod DR Runbooks and Game-Day Exercises

Guide

Revision 1

On this page

Scope#

Implemented disaster-recovery runbooks and game-day exercise automation for RunPod provider-failure and endpoint-region-failure incident classes.

Implemented Artifacts#

  • Endpoint region failover drill automation:
    • scripts/isis/run_endpoint_region_failover_drill.sh
  • Composite DR game-day harness:
    • scripts/isis/run_runpod_disaster_recovery_gameday.sh
  • Runbooks:
    • docs/domains/isis/runbooks/provider-outage-drill.md
    • docs/domains/isis/runbooks/endpoint-region-failover-drill.md
    • docs/domains/isis/runbooks/runpod-dr-gameday.md
    • docs/domains/isis/runbooks/runpod-oncall.md
  • Infra/process documentation:
    • docs/infrastructure/runpod-integration.md
    • docs/reference/ci-cd.md

Drill Model#

Provider Failure#

  • Uses existing run_provider_outage_drill.sh to simulate RunPod routing outage and validate fallback to RunComfy.

Endpoint Region Failure#

  • Uses run_endpoint_region_failover_drill.sh to simulate regional endpoint loss by rerouting stable endpoint traffic to the configured canary endpoint for a selected endpoint family.
  • Verifies post-mutation routing convergence and baseline restoration.

Composite Game-Day#

  • Uses run_runpod_disaster_recovery_gameday.sh to orchestrate both scenarios, aggregate scenario outcomes, and emit a unified report artifact.

Verification#

bash
bash -n scripts/isis/run_provider_outage_drill.sh scripts/isis/run_endpoint_region_failover_drill.sh scripts/isis/run_runpod_disaster_recovery_gameday.sh
STRICT=false DRY_RUN=true RUN_PROVIDER_FAILURE_SCENARIO=false RUN_REGION_FAILURE_SCENARIO=false scripts/isis/run_runpod_disaster_recovery_gameday.sh

Outcome#

WS09-010 is now closed with deterministic disaster-recovery runbooks and repeatable game-day automation for both provider outage and endpoint region failover scenarios.