# OPS-003 Certification: Provider Outage Disaster-Recovery Drill

Date: 2026-02-22  
Checklist Item: `OPS-003`  
Ticket: `ISIS-0238`

## Scope

Operational resilience coverage for provider outage scenarios where RunPod is
unavailable:

1. Deterministic outage simulation against Isis provider routing feature flags.
2. Fallback-path verification against provider operations telemetry.
3. Synthetic generation verification under fallback routing.
4. Deterministic restoration and post-restore verification.

## Evidence Matrix

| Requirement                                       | Evidence                                                                                                                                                                                                           | Status |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ |
| Provider outage drill runbook exists              | `docs/domains/isis/runbooks/provider-outage-drill.md` defines prerequisites, dry-run and execution procedures, success criteria, failure handling, cadence, and evidence artifacts.                                | Pass   |
| Outage simulation is automatable                  | `scripts/isis/run_provider_outage_drill.sh` applies `COMFYUI_PROVIDER=runcomfy`, `ENABLE_RUNPOD_MIGRATION=false`, and `RUNPOD_MIGRATION_PERCENTAGE=0` in writable environments.                                    | Pass   |
| Fallback verification is explicit and testable    | Same script verifies post-drill `/api/v1/jobs/feature-flags` routing state and `/api/v1/jobs/providers/operations` fallback health before marking drill pass.                                                      | Pass   |
| Synthetic drill traffic is validated              | Same script can submit/poll a synthetic job (`SUBMIT_SYNTHETIC_JOB=true`) and records acceptance and progress checks in machine-readable report output.                                                            | Pass   |
| Restoration and restore verification are codified | Same script restores baseline routing flags and verifies restored state matches the pre-drill baseline snapshot; failures are recorded in `cleanup.restore_*` checks.                                              | Pass   |
| On-call workflow references drill artifact        | `docs/domains/isis/runbooks/runpod-oncall.md` now links `docs/domains/isis/runbooks/provider-outage-drill.md` under related documentation to ensure outage drills are part of ongoing operational readiness flows. | Pass   |

## Reproduction Commands

```bash
bash -n scripts/isis/run_provider_outage_drill.sh

DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh

DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh
```
