# Isis Provider Outage Disaster Recovery Drill Runbook

Last Updated: 2026-02-22  
Scope: RunPod outage simulation with RunComfy fallback verification  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`

## Purpose

Run a repeatable disaster-recovery drill for RunPod provider outage scenarios
and verify that Isis can fallback to non-RunPod provider routing without
breaking generation flows.

## Drill Objectives

1. Simulate RunPod outage conditions via routing controls.
2. Verify fallback routing (`runcomfy`) is active and healthy.
3. Verify synthetic generation submissions still succeed.
4. Restore original routing state after drill completion.

## Preconditions

- Execute in non-production environment where provider feature flags are
  writable.
- `ISIS_API_BASE_URL` and privileged `ISIS_ADMIN_TOKEN` available.
- RunComfy fallback provider is configured and healthy.
- Change ticket opened with drill window and rollback owner.

## Drill Procedure

### A. Dry-Run Validation (No Mutations)

```bash
DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh
```

Dry-run captures baseline snapshots only and marks mutation-dependent checks as
`skip`.

### B. Execute Outage Drill

```bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_provider_outage_drill.sh
```

### C. What the Drill Automation Does

`scripts/isis/run_provider_outage_drill.sh` performs:

1. Capture baseline `/api/v1/jobs/feature-flags`.
2. Capture baseline `/api/v1/jobs/providers/operations`.
3. Apply outage simulation flags:
   - `COMFYUI_PROVIDER=runcomfy`
   - `ENABLE_RUNPOD_MIGRATION=false`
   - `RUNPOD_MIGRATION_PERCENTAGE=0`
4. Verify routing fallback flags.
5. Verify `/api/v1/jobs/providers/operations` shows fallback provider health.
6. Submit and poll synthetic drill job (optional).
7. Restore baseline flags (if `RESTORE_FLAGS=true`).
8. Verify restored flags match baseline.
9. Emit JSON drill report in `tmp/isis-provider-drill/`.

### D. Optional Environment Controls

- `API_TIMEOUT_SECONDS` (default `30`) controls HTTP timeout for drill calls.
- `SUBMIT_SYNTHETIC_JOB=false` skips synthetic submit/progress checks.
- `RESTORE_FLAGS=false` disables restore for controlled experiments (not
  recommended for routine drills).
- `STRICT=true` makes the script exit non-zero if any check fails.

## Success Criteria

Drill passes only if:

- fallback flags are applied as expected during simulation.
- provider operations indicate healthy fallback path.
- synthetic job is accepted and reaches expected non-failed state.
- original routing flags are restored successfully.
- restored routing flags are verified equal to pre-drill baseline.

## Failure Handling

If drill fails:

1. Immediately restore baseline feature flags.
2. Validate provider operations and runpod operations snapshots.
3. Open incident if fallback path is degraded.
4. File remediation tasks before next drill.

## Evidence Artifacts

- `tmp/isis-provider-drill/provider-outage-drill-*.json`
- baseline/post-drill feature flag snapshots
- baseline/post-drill provider operations snapshots
- restored feature-flag verification snapshot
- synthetic submit/status snapshots

## Recommended Cadence

- Minimum: monthly in staging.
- Additional runs:
  - before major provider routing changes
  - after RunPod migration/canary logic changes
  - after major incident involving provider instability.

## Related Artifacts

- `scripts/isis/run_provider_outage_drill.sh`
- `scripts/isis/run_runpod_disaster_recovery_gameday.sh`
- `docs/domains/isis/runbooks/runpod-oncall.md`
- `docs/domains/isis/runbooks/endpoint-region-failover-drill.md`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md`
