# ISIS v2 WS14-010: SLO-Based Release Gates and Automated Rollback Triggers

## Scope

`WS14-010` implements an executable release-gate evaluator for RunPod operations
that enforces SLO/anomaly/synthetic/canary/guardrail checks and can
automatically trigger rollback execution when release criteria fail.

## Research and Existing Implementation Alignment

Before implementation, existing canary and rollback automation was reviewed:

- Existing canary SLO/cost verification automation:
  - `scripts/isis/verify_canary_slo_cost.sh`
- Existing RunPod endpoint canary strategy execution and rollback callbacks:
  - `scripts/isis/execute_runpod_endpoint_canary_strategy.sh`
- Existing release gameday operations flow:
  - `scripts/isis/run_release_gameday.sh`
- Existing RunPod strategy and rollback runbooks:
  - `docs/domains/isis/remediation-v2/ISIS_V2_RUNPOD_ENDPOINT_CANARY_STRATEGY_AND_ROLLBACK.md`
  - `docs/domains/isis/remediation-v2/ISIS_V2_ROLLBACK_AND_FREEZE_POLICY.md`
  - `docs/domains/isis/runbooks/runpod-oncall.md`

This task intentionally extends those workflows by introducing a deterministic
release-gate decision artifact and optional rollback execution handoff.

## Deliverables

1. Added release-gate evaluation automation:

- `scripts/isis/evaluate-runpod-release-gate.mjs`

Script capabilities:

- reads RunPod operations snapshot from `--operations-file` or
  `--operations-url`,
- evaluates release checks across:
  - `runpod.operations.available`,
  - `slo.endpoint.breaches`,
  - `anomaly.release.blockers`,
  - `synthetic.transactions.gate`,
  - `canary.strategy.rollback`,
  - `budget.guardrail.mode`,
- emits a structured release-gate report JSON with check-level details,
- supports non-zero failure behavior via `--fail-on-breach=true`,
- supports rollback automation via `--auto-rollback=true`, delegating execution
  to `scripts/isis/execute_runpod_endpoint_canary_strategy.sh`,
- supports optional rollback callback pass-through with
  `--rollback-callback=<command>`.

2. Added deterministic fixture inputs for gate validation:

- `scripts/isis/fixtures/release-gate-operations-pass.json`
- `scripts/isis/fixtures/release-gate-operations-fail.json`

Fixtures cover both passing and failing operational conditions and enable
repeatable gate/rollback validation.

## Verification Commands

```bash
node --check scripts/isis/evaluate-runpod-release-gate.mjs
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-pass.json --report-file=tmp/isis-release-gate/pass-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report-auto-rollback.json --auto-rollback=true --fail-on-breach=false
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

## Verification Notes

- Passing fixture produces an all-green release-gate report and zero exit when
  `--fail-on-breach=true`.
- Failing fixture produces gate failures and exit code `2` when
  `--fail-on-breach=true`.
- Failing fixture with `--auto-rollback=true` executes endpoint canary strategy
  rollback flow and emits rollback execution evidence in report output.
