Eve library

Isis Release Failure Game-Day Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: WS18-003 failure scenarios for provider failure, queue saturation, and bad deploy rollback.

Purpose#

Run a deterministic release-focused game-day harness that validates Isis readiness for the three highest-risk launch failure classes:

  1. Provider failure fallback behavior
  2. Queue saturation detection and classification
  3. Bad deploy rollback decisioning

Script#

  • scripts/isis/run_release_failure_gameday.sh

Default outputs:

  • tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.json
  • tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.md

Scenario Matrix#

Scenario ID Failure Class Implementation Path Pass Condition
GDS-001 Provider failure Provider outage drill evidence (run_provider_outage_drill.sh) .overall.pass=true in provider drill report
GDS-002 Queue saturation Queue indicator evaluation from API, fixture, or synthetic injection saturation/starvation pressure signals detected
GDS-003 Bad deploy rollback Forced rollback decision simulation via advance_queue_canary_rollout.sh decision action is rollback and target is 0%

Usage#

Deterministic Dry Scenario Set (No Live API Required)#

bash
RUN_PROVIDER_FAILURE_SCENARIO=false \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh

Live Provider Failure + API Queue Signals#

bash
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_PROVIDER_DRILL=true \
PROVIDER_DRILL_DRY_RUN=false \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh

Replay With Existing Provider Drill Artifact#

bash
PROVIDER_FAILURE_REPORT_FILE=tmp/isis-provider-drill/provider-outage-drill-<timestamp>.json \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh

Key Inputs#

  • RUN_PROVIDER_FAILURE_SCENARIO
  • RUN_QUEUE_SATURATION_SCENARIO
  • RUN_BAD_DEPLOY_ROLLBACK_SCENARIO
  • PROVIDER_FAILURE_REPORT_FILE
  • QUEUE_SATURATION_SOURCE (auto|api|fixture|synthetic)
  • QUEUE_SATURATION_REPORT_FILE
  • ISIS_API_BASE_URL, ISIS_ADMIN_TOKEN
  • BAD_DEPLOY_DECISION_FILE
  • BAD_DEPLOY_CURRENT_PERCENTAGE

Exit Semantics#

  • STRICT=false (default): always emits reports; returns non-zero only for runtime/script errors.
  • STRICT=true: returns exit 2 when any required scenario is not pass.
  • scripts/isis/run_provider_outage_drill.sh
  • scripts/isis/advance_queue_canary_rollout.sh
  • docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md
  • docs/domains/isis/runbooks/provider-outage-drill.md
  • docs/domains/isis/runbooks/runpod-oncall.md