# ISIS v2 WS09-010: RunPod DR Runbooks and Game-Day Exercises

## Scope

Implemented disaster-recovery runbooks and game-day exercise automation for
RunPod provider-failure and endpoint-region-failure incident classes.

## Implemented Artifacts

- Endpoint region failover drill automation:
  - `scripts/isis/run_endpoint_region_failover_drill.sh`
- Composite DR game-day harness:
  - `scripts/isis/run_runpod_disaster_recovery_gameday.sh`
- Runbooks:
  - `docs/domains/isis/runbooks/provider-outage-drill.md`
  - `docs/domains/isis/runbooks/endpoint-region-failover-drill.md`
  - `docs/domains/isis/runbooks/runpod-dr-gameday.md`
  - `docs/domains/isis/runbooks/runpod-oncall.md`
- Infra/process documentation:
  - `docs/infrastructure/runpod-integration.md`
  - `docs/reference/ci-cd.md`

## Drill Model

### Provider Failure

- Uses existing `run_provider_outage_drill.sh` to simulate RunPod routing outage
  and validate fallback to RunComfy.

### Endpoint Region Failure

- Uses `run_endpoint_region_failover_drill.sh` to simulate regional endpoint
  loss by rerouting stable endpoint traffic to the configured canary endpoint
  for a selected endpoint family.
- Verifies post-mutation routing convergence and baseline restoration.

### Composite Game-Day

- Uses `run_runpod_disaster_recovery_gameday.sh` to orchestrate both scenarios,
  aggregate scenario outcomes, and emit a unified report artifact.

## Verification

```bash
bash -n scripts/isis/run_provider_outage_drill.sh scripts/isis/run_endpoint_region_failover_drill.sh scripts/isis/run_runpod_disaster_recovery_gameday.sh
STRICT=false DRY_RUN=true RUN_PROVIDER_FAILURE_SCENARIO=false RUN_REGION_FAILURE_SCENARIO=false scripts/isis/run_runpod_disaster_recovery_gameday.sh
```

## Outcome

WS09-010 is now closed with deterministic disaster-recovery runbooks and
repeatable game-day automation for both provider outage and endpoint region
failover scenarios.
