# Isis Release Failure Game-Day Runbook

Last Updated: 2026-02-24  
Scope: WS18-003 failure scenarios for provider failure, queue saturation, and
bad deploy rollback.

## Purpose

Run a deterministic release-focused game-day harness that validates Isis
readiness for the three highest-risk launch failure classes:

1. Provider failure fallback behavior
2. Queue saturation detection and classification
3. Bad deploy rollback decisioning

## Script

- `scripts/isis/run_release_failure_gameday.sh`

Default outputs:

- `tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.json`
- `tmp/isis-release-failure-gameday/release-failure-gameday-<timestamp>.md`

## Scenario Matrix

| Scenario ID | Failure Class       | Implementation Path                                                       | Pass Condition                                   |
| ----------- | ------------------- | ------------------------------------------------------------------------- | ------------------------------------------------ |
| `GDS-001`   | Provider failure    | Provider outage drill evidence (`run_provider_outage_drill.sh`)           | `.overall.pass=true` in provider drill report    |
| `GDS-002`   | Queue saturation    | Queue indicator evaluation from API, fixture, or synthetic injection      | saturation/starvation pressure signals detected  |
| `GDS-003`   | Bad deploy rollback | Forced rollback decision simulation via `advance_queue_canary_rollout.sh` | decision action is `rollback` and target is `0%` |

## Usage

### Deterministic Dry Scenario Set (No Live API Required)

```bash
RUN_PROVIDER_FAILURE_SCENARIO=false \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
```

### Live Provider Failure + API Queue Signals

```bash
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="..." \
RUN_PROVIDER_FAILURE_SCENARIO=true \
RUN_PROVIDER_DRILL=true \
PROVIDER_DRILL_DRY_RUN=false \
QUEUE_SATURATION_SOURCE=api \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
```

### Replay With Existing Provider Drill Artifact

```bash
PROVIDER_FAILURE_REPORT_FILE=tmp/isis-provider-drill/provider-outage-drill-<timestamp>.json \
QUEUE_SATURATION_SOURCE=synthetic \
STRICT=true \
scripts/isis/run_release_failure_gameday.sh
```

## Key Inputs

- `RUN_PROVIDER_FAILURE_SCENARIO`
- `RUN_QUEUE_SATURATION_SCENARIO`
- `RUN_BAD_DEPLOY_ROLLBACK_SCENARIO`
- `PROVIDER_FAILURE_REPORT_FILE`
- `QUEUE_SATURATION_SOURCE` (`auto|api|fixture|synthetic`)
- `QUEUE_SATURATION_REPORT_FILE`
- `ISIS_API_BASE_URL`, `ISIS_ADMIN_TOKEN`
- `BAD_DEPLOY_DECISION_FILE`
- `BAD_DEPLOY_CURRENT_PERCENTAGE`

## Exit Semantics

- `STRICT=false` (default): always emits reports; returns non-zero only for
  runtime/script errors.
- `STRICT=true`: returns exit `2` when any required scenario is not `pass`.

## Related Artifacts

- `scripts/isis/run_provider_outage_drill.sh`
- `scripts/isis/advance_queue_canary_rollout.sh`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`
- `docs/domains/isis/runbooks/provider-outage-drill.md`
- `docs/domains/isis/runbooks/runpod-oncall.md`
