Scope#
WS14-010 implements an executable release-gate evaluator for RunPod operations
that enforces SLO/anomaly/synthetic/canary/guardrail checks and can
automatically trigger rollback execution when release criteria fail.
Research and Existing Implementation Alignment#
Before implementation, existing canary and rollback automation was reviewed:
- Existing canary SLO/cost verification automation:
scripts/isis/verify_canary_slo_cost.sh
- Existing RunPod endpoint canary strategy execution and rollback callbacks:
scripts/isis/execute_runpod_endpoint_canary_strategy.sh
- Existing release gameday operations flow:
scripts/isis/run_release_gameday.sh
- Existing RunPod strategy and rollback runbooks:
docs/domains/isis/remediation-v2/ISIS_V2_RUNPOD_ENDPOINT_CANARY_STRATEGY_AND_ROLLBACK.mddocs/domains/isis/remediation-v2/ISIS_V2_ROLLBACK_AND_FREEZE_POLICY.mddocs/domains/isis/runbooks/runpod-oncall.md
This task intentionally extends those workflows by introducing a deterministic release-gate decision artifact and optional rollback execution handoff.
Deliverables#
- Added release-gate evaluation automation:
scripts/isis/evaluate-runpod-release-gate.mjs
Script capabilities:
- reads RunPod operations snapshot from
--operations-fileor--operations-url, - evaluates release checks across:
runpod.operations.available,slo.endpoint.breaches,anomaly.release.blockers,synthetic.transactions.gate,canary.strategy.rollback,budget.guardrail.mode,
- emits a structured release-gate report JSON with check-level details,
- supports non-zero failure behavior via
--fail-on-breach=true, - supports rollback automation via
--auto-rollback=true, delegating execution toscripts/isis/execute_runpod_endpoint_canary_strategy.sh, - supports optional rollback callback pass-through with
--rollback-callback=<command>.
- Added deterministic fixture inputs for gate validation:
scripts/isis/fixtures/release-gate-operations-pass.jsonscripts/isis/fixtures/release-gate-operations-fail.json
Fixtures cover both passing and failing operational conditions and enable repeatable gate/rollback validation.
Verification Commands#
bash
node --check scripts/isis/evaluate-runpod-release-gate.mjs
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-pass.json --report-file=tmp/isis-release-gate/pass-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report-auto-rollback.json --auto-rollback=true --fail-on-breach=false
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
- Passing fixture produces an all-green release-gate report and zero exit when
--fail-on-breach=true. - Failing fixture produces gate failures and exit code
2when--fail-on-breach=true. - Failing fixture with
--auto-rollback=trueexecutes endpoint canary strategy rollback flow and emits rollback execution evidence in report output.