Last Updated: 2026-02-24
Scope: Composite game-day for provider-failure and endpoint-region-failure
scenarios
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team
Purpose#
Execute a deterministic game-day that validates Isis RunPod disaster-recovery readiness for:
- RunPod provider-level failure (fallback to RunComfy).
- Endpoint-family regional failure (canary-to-stable failover routing).
Scenario Matrix#
| Scenario ID | Failure Class | Automation Script | Outcome Gate |
|---|---|---|---|
| DR-GD-001 | Provider failure | scripts/isis/run_provider_outage_drill.sh |
overall.pass=true |
| DR-GD-002 | Endpoint regional failover | scripts/isis/run_endpoint_region_failover_drill.sh |
overall.pass=true |
Preconditions#
- Non-production environment with writable provider feature flags.
ISIS_API_BASE_URLand privilegedISIS_ADMIN_TOKEN.- RunComfy fallback provider configured and healthy.
- Target endpoint families have canary endpoints provisioned.
- Incident commander and rollback owner assigned.
Composite Execution#
Dry run#
bash
DRY_RUN=true \
STRICT=true \
REGION_FAMILIES="comfyui,flux" \
scripts/isis/run_runpod_disaster_recovery_gameday.sh
Live game-day#
bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_REGION_FAILURE_SCENARIO=true \
REGION_FAMILIES="comfyui,flux" \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_runpod_disaster_recovery_gameday.sh
Key Controls#
RESTORE_FLAGS=true: mandatory for routine exercises.SUBMIT_SYNTHETIC_JOB=true: validates runtime path under injected failures.REGION_FAMILIES: comma-separated endpoint families for regional failover drills.STRICT=true: exits non-zero if any scenario fails.
Success Criteria#
Game-day passes only when:
- each enabled scenario emits a valid report,
- no scenario reports failed checks,
- routing flags are restored to baseline,
- synthetic validation succeeds or is explicitly skipped with approval.
Evidence Artifacts#
- Composite report:
tmp/isis-runpod-dr-gameday/runpod-dr-gameday-*.json
- Scenario reports:
tmp/isis-runpod-dr-gameday/provider-outage-drill.jsontmp/isis-runpod-dr-gameday/region-failover-<family>.json
Cadence#
- Monthly in staging.
- Mandatory before major endpoint topology or provider-routing changes.
- Mandatory after SEV-1/SEV-2 incidents in provider routing.
Related Artifacts#
scripts/isis/run_runpod_disaster_recovery_gameday.shscripts/isis/run_provider_outage_drill.shscripts/isis/run_endpoint_region_failover_drill.shdocs/domains/isis/runbooks/provider-outage-drill.mddocs/domains/isis/runbooks/endpoint-region-failover-drill.mddocs/domains/isis/runbooks/runpod-oncall.md