Date: 2026-02-22
Checklist Item: OPS-003
Ticket: ISIS-0238
Scope#
Operational resilience coverage for provider outage scenarios where RunPod is unavailable:
- Deterministic outage simulation against Isis provider routing feature flags.
- Fallback-path verification against provider operations telemetry.
- Synthetic generation verification under fallback routing.
- Deterministic restoration and post-restore verification.
Evidence Matrix#
| Requirement | Evidence | Status |
|---|---|---|
| Provider outage drill runbook exists | docs/domains/isis/runbooks/provider-outage-drill.md defines prerequisites, dry-run and execution procedures, success criteria, failure handling, cadence, and evidence artifacts. |
Pass |
| Outage simulation is automatable | scripts/isis/run_provider_outage_drill.sh applies COMFYUI_PROVIDER=runcomfy, ENABLE_RUNPOD_MIGRATION=false, and RUNPOD_MIGRATION_PERCENTAGE=0 in writable environments. |
Pass |
| Fallback verification is explicit and testable | Same script verifies post-drill /api/v1/jobs/feature-flags routing state and /api/v1/jobs/providers/operations fallback health before marking drill pass. |
Pass |
| Synthetic drill traffic is validated | Same script can submit/poll a synthetic job (SUBMIT_SYNTHETIC_JOB=true) and records acceptance and progress checks in machine-readable report output. |
Pass |
| Restoration and restore verification are codified | Same script restores baseline routing flags and verifies restored state matches the pre-drill baseline snapshot; failures are recorded in cleanup.restore_* checks. |
Pass |
| On-call workflow references drill artifact | docs/domains/isis/runbooks/runpod-oncall.md now links docs/domains/isis/runbooks/provider-outage-drill.md under related documentation to ensure outage drills are part of ongoing operational readiness flows. |
Pass |
Reproduction Commands#
bash
bash -n scripts/isis/run_provider_outage_drill.sh
DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh