Last Updated: 2026-02-24
Scope: WS18-003 failure scenarios for provider failure, queue saturation, and
bad deploy rollback.
Purpose#
Run a deterministic release-focused game-day harness that validates Isis readiness for the three highest-risk launch failure classes:
- Provider failure fallback behavior
- Queue saturation detection and classification
- Bad deploy rollback decisioning
Script#
scripts/isis/run_release_failure_gameday.sh
Default outputs:
tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.jsontmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.md
Scenario Matrix#
| Scenario ID | Failure Class | Implementation Path | Pass Condition |
|---|---|---|---|
GDS-001 |
Provider failure | Provider outage drill evidence (run_provider_outage_drill.sh) |
.overall.pass=true in provider drill report |
GDS-002 |
Queue saturation | Queue indicator evaluation from API, fixture, or synthetic injection | saturation/starvation pressure signals detected |
GDS-003 |
Bad deploy rollback | Forced rollback decision simulation via advance_queue_canary_rollout.sh |
decision action is rollback and target is 0% |
Usage#
Deterministic Dry Scenario Set (No Live API Required)#
bash
RUN_PROVIDER_FAILURE_SCENARIO=false \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
Live Provider Failure + API Queue Signals#
bash
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_PROVIDER_DRILL=true \
PROVIDER_DRILL_DRY_RUN=false \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
Replay With Existing Provider Drill Artifact#
bash
PROVIDER_FAILURE_REPORT_FILE=tmp/isis-provider-drill/provider-outage-drill-<timestamp>.json \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
Key Inputs#
RUN_PROVIDER_FAILURE_SCENARIORUN_QUEUE_SATURATION_SCENARIORUN_BAD_DEPLOY_ROLLBACK_SCENARIOPROVIDER_FAILURE_REPORT_FILEQUEUE_SATURATION_SOURCE(auto|api|fixture|synthetic)QUEUE_SATURATION_REPORT_FILEISIS_API_BASE_URL,ISIS_ADMIN_TOKENBAD_DEPLOY_DECISION_FILEBAD_DEPLOY_CURRENT_PERCENTAGE
Exit Semantics#
STRICT=false(default): always emits reports; returns non-zero only for runtime/script errors.STRICT=true: returns exit2when any required scenario is notpass.
Related Artifacts#
scripts/isis/run_provider_outage_drill.shscripts/isis/advance_queue_canary_rollout.shdocs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.mddocs/domains/isis/runbooks/provider-outage-drill.mddocs/domains/isis/runbooks/runpod-oncall.md