Scope#
Implemented disaster-recovery runbooks and game-day exercise automation for RunPod provider-failure and endpoint-region-failure incident classes.
Implemented Artifacts#
- Endpoint region failover drill automation:
scripts/isis/run_endpoint_region_failover_drill.sh
- Composite DR game-day harness:
scripts/isis/run_runpod_disaster_recovery_gameday.sh
- Runbooks:
docs/domains/isis/runbooks/provider-outage-drill.mddocs/domains/isis/runbooks/endpoint-region-failover-drill.mddocs/domains/isis/runbooks/runpod-dr-gameday.mddocs/domains/isis/runbooks/runpod-oncall.md
- Infra/process documentation:
docs/infrastructure/runpod-integration.mddocs/reference/ci-cd.md
Drill Model#
Provider Failure#
- Uses existing
run_provider_outage_drill.shto simulate RunPod routing outage and validate fallback to RunComfy.
Endpoint Region Failure#
- Uses
run_endpoint_region_failover_drill.shto simulate regional endpoint loss by rerouting stable endpoint traffic to the configured canary endpoint for a selected endpoint family. - Verifies post-mutation routing convergence and baseline restoration.
Composite Game-Day#
- Uses
run_runpod_disaster_recovery_gameday.shto orchestrate both scenarios, aggregate scenario outcomes, and emit a unified report artifact.
Verification#
bash
bash -n scripts/isis/run_provider_outage_drill.sh scripts/isis/run_endpoint_region_failover_drill.sh scripts/isis/run_runpod_disaster_recovery_gameday.sh
STRICT=false DRY_RUN=true RUN_PROVIDER_FAILURE_SCENARIO=false RUN_REGION_FAILURE_SCENARIO=false scripts/isis/run_runpod_disaster_recovery_gameday.sh
Outcome#
WS09-010 is now closed with deterministic disaster-recovery runbooks and repeatable game-day automation for both provider outage and endpoint region failover scenarios.