Eve library

OPS-003 Certification: Provider Outage Disaster-Recovery Drill

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: OPS-003
Ticket: ISIS-0238

Scope#

Operational resilience coverage for provider outage scenarios where RunPod is unavailable:

  1. Deterministic outage simulation against Isis provider routing feature flags.
  2. Fallback-path verification against provider operations telemetry.
  3. Synthetic generation verification under fallback routing.
  4. Deterministic restoration and post-restore verification.

Evidence Matrix#

Requirement Evidence Status
Provider outage drill runbook exists docs/domains/isis/runbooks/provider-outage-drill.md defines prerequisites, dry-run and execution procedures, success criteria, failure handling, cadence, and evidence artifacts. Pass
Outage simulation is automatable scripts/isis/run_provider_outage_drill.sh applies COMFYUI_PROVIDER=runcomfy, ENABLE_RUNPOD_MIGRATION=false, and RUNPOD_MIGRATION_PERCENTAGE=0 in writable environments. Pass
Fallback verification is explicit and testable Same script verifies post-drill /api/v1/jobs/feature-flags routing state and /api/v1/jobs/providers/operations fallback health before marking drill pass. Pass
Synthetic drill traffic is validated Same script can submit/poll a synthetic job (SUBMIT_SYNTHETIC_JOB=true) and records acceptance and progress checks in machine-readable report output. Pass
Restoration and restore verification are codified Same script restores baseline routing flags and verifies restored state matches the pre-drill baseline snapshot; failures are recorded in cleanup.restore_* checks. Pass
On-call workflow references drill artifact docs/domains/isis/runbooks/runpod-oncall.md now links docs/domains/isis/runbooks/provider-outage-drill.md under related documentation to ensure outage drills are part of ongoing operational readiness flows. Pass

Reproduction Commands#

bash
bash -n scripts/isis/run_provider_outage_drill.sh

DRY_RUN=true \
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh

DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/run_provider_outage_drill.sh