Last Updated: 2026-02-24
Scope: Endpoint-family regional failover simulation using stable/canary routing
controls
Primary Owners: @oshun/devops, @oshun/isis-team
Purpose#
Run a repeatable disaster-recovery drill that simulates regional endpoint loss by promoting the configured canary endpoint to the stable route for a target endpoint family, then verifies routing, synthetic execution, and restoration.
Drill Objectives#
- Validate every critical endpoint family has a secondary (canary) route.
- Simulate regional failover by rerouting stable traffic to the canary endpoint.
- Verify provider operations and runpod operations snapshots converge to expected failover state.
- Restore feature flags to baseline and verify parity.
Preconditions#
- Execute in non-production where provider feature flags are writable.
ISIS_API_BASE_URLand privilegedISIS_ADMIN_TOKENare available.- Target family has both stable and canary endpoint IDs configured.
- Change ticket is opened with rollback owner and drill window.
Execute Drill#
Dry run (no mutations)#
bash
DRY_RUN=true \
STRICT=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh
Live simulation (with automatic restore)#
bash
DRY_RUN=false \
STRICT=true \
RESTORE_FLAGS=true \
SUBMIT_SYNTHETIC_JOB=true \
TARGET_FAMILY=comfyui \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
scripts/isis/run_endpoint_region_failover_drill.sh
Supported Families#
comfyuisdfluxcharacter-consistencycinematic-videothree-dtextureaudio-sfxaudio-voiceaudio-music
What Automation Verifies#
- Baseline feature-flag snapshot capture.
- Baseline stable/canary endpoint presence for target family.
- Failover mutation:
- stable endpoint flag set to canary endpoint ID
- canary percentage set to
0
- Post-mutation convergence in
/api/v1/jobs/runpod/operations. - Optional synthetic job acceptance/progress checks.
- Full restore of stable endpoint + canary percentage flags.
- Restore verification parity with baseline.
Success Criteria#
Drill passes only if:
- failover mutation succeeds,
- active stable endpoint equals expected failover endpoint,
- synthetic flow remains non-failed (when enabled),
- baseline routing state is restored and verified.
Evidence Artifacts#
tmp/isis-region-drill/endpoint-region-failover-drill-*.json- feature-flag baseline/post/restore snapshots
- runpod operations post-drill snapshot
- synthetic submit/status snapshots (if enabled)
Related Artifacts#
scripts/isis/run_endpoint_region_failover_drill.shdocs/domains/isis/runbooks/provider-outage-drill.mddocs/domains/isis/runbooks/runpod-oncall.md