# Isis RunPod Disaster Recovery Game-Day Runbook

Last Updated: 2026-02-24  
Scope: Composite game-day for provider-failure and endpoint-region-failure
scenarios  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`, `@oshun/ml-team`

## Purpose

Execute a deterministic game-day that validates Isis RunPod disaster-recovery
readiness for:

1. RunPod provider-level failure (fallback to RunComfy).
2. Endpoint-family regional failure (canary-to-stable failover routing).

## Scenario Matrix

| Scenario ID | Failure Class              | Automation Script                                    | Outcome Gate        |
| ----------- | -------------------------- | ---------------------------------------------------- | ------------------- |
| DR-GD-001   | Provider failure           | `scripts/isis/run_provider_outage_drill.sh`          | `overall.pass=true` |
| DR-GD-002   | Endpoint regional failover | `scripts/isis/run_endpoint_region_failover_drill.sh` | `overall.pass=true` |

## Preconditions

- Non-production environment with writable provider feature flags.
- `ISIS_API_BASE_URL` and privileged `ISIS_ADMIN_TOKEN`.
- RunComfy fallback provider configured and healthy.
- Target endpoint families have canary endpoints provisioned.
- Incident commander and rollback owner assigned.

## Composite Execution

### Dry run

```bash
DRY_RUN=true \
STRICT=true \
REGION_FAMILIES="comfyui,flux" \
scripts/isis/run_runpod_disaster_recovery_gameday.sh
```

### Live game-day

```bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_REGION_FAILURE_SCENARIO=true \
REGION_FAMILIES="comfyui,flux" \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_runpod_disaster_recovery_gameday.sh
```

## Key Controls

- `RESTORE_FLAGS=true`: mandatory for routine exercises.
- `SUBMIT_SYNTHETIC_JOB=true`: validates runtime path under injected failures.
- `REGION_FAMILIES`: comma-separated endpoint families for regional failover
  drills.
- `STRICT=true`: exits non-zero if any scenario fails.

## Success Criteria

Game-day passes only when:

- each enabled scenario emits a valid report,
- no scenario reports failed checks,
- routing flags are restored to baseline,
- synthetic validation succeeds or is explicitly skipped with approval.

## Evidence Artifacts

- Composite report:
  - `tmp/isis-runpod-dr-gameday/runpod-dr-gameday-*.json`
- Scenario reports:
  - `tmp/isis-runpod-dr-gameday/provider-outage-drill.json`
  - `tmp/isis-runpod-dr-gameday/region-failover-<family>.json`

## Cadence

- Monthly in staging.
- Mandatory before major endpoint topology or provider-routing changes.
- Mandatory after SEV-1/SEV-2 incidents in provider routing.

## Related Artifacts

- `scripts/isis/run_runpod_disaster_recovery_gameday.sh`
- `scripts/isis/run_provider_outage_drill.sh`
- `scripts/isis/run_endpoint_region_failover_drill.sh`
- `docs/domains/isis/runbooks/provider-outage-drill.md`
- `docs/domains/isis/runbooks/endpoint-region-failover-drill.md`
- `docs/domains/isis/runbooks/runpod-oncall.md`
