Eve library

ISIS v2 WS14-010: SLO-Based Release Gates and Automated Rollback Triggers

Guide

Revision 1

On this page

Scope#

WS14-010 implements an executable release-gate evaluator for RunPod operations that enforces SLO/anomaly/synthetic/canary/guardrail checks and can automatically trigger rollback execution when release criteria fail.

Research and Existing Implementation Alignment#

Before implementation, existing canary and rollback automation was reviewed:

  • Existing canary SLO/cost verification automation:
    • scripts/isis/verify_canary_slo_cost.sh
  • Existing RunPod endpoint canary strategy execution and rollback callbacks:
    • scripts/isis/execute_runpod_endpoint_canary_strategy.sh
  • Existing release gameday operations flow:
    • scripts/isis/run_release_gameday.sh
  • Existing RunPod strategy and rollback runbooks:
    • docs/domains/isis/remediation-v2/ISIS_V2_RUNPOD_ENDPOINT_CANARY_STRATEGY_AND_ROLLBACK.md
    • docs/domains/isis/remediation-v2/ISIS_V2_ROLLBACK_AND_FREEZE_POLICY.md
    • docs/domains/isis/runbooks/runpod-oncall.md

This task intentionally extends those workflows by introducing a deterministic release-gate decision artifact and optional rollback execution handoff.

Deliverables#

  1. Added release-gate evaluation automation:
  • scripts/isis/evaluate-runpod-release-gate.mjs

Script capabilities:

  • reads RunPod operations snapshot from --operations-file or --operations-url,
  • evaluates release checks across:
    • runpod.operations.available,
    • slo.endpoint.breaches,
    • anomaly.release.blockers,
    • synthetic.transactions.gate,
    • canary.strategy.rollback,
    • budget.guardrail.mode,
  • emits a structured release-gate report JSON with check-level details,
  • supports non-zero failure behavior via --fail-on-breach=true,
  • supports rollback automation via --auto-rollback=true, delegating execution to scripts/isis/execute_runpod_endpoint_canary_strategy.sh,
  • supports optional rollback callback pass-through with --rollback-callback=<command>.
  1. Added deterministic fixture inputs for gate validation:
  • scripts/isis/fixtures/release-gate-operations-pass.json
  • scripts/isis/fixtures/release-gate-operations-fail.json

Fixtures cover both passing and failing operational conditions and enable repeatable gate/rollback validation.

Verification Commands#

bash
node --check scripts/isis/evaluate-runpod-release-gate.mjs
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-pass.json --report-file=tmp/isis-release-gate/pass-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report.json --fail-on-breach=true
node scripts/isis/evaluate-runpod-release-gate.mjs --operations-file=scripts/isis/fixtures/release-gate-operations-fail.json --report-file=tmp/isis-release-gate/fail-report-auto-rollback.json --auto-rollback=true --fail-on-breach=false
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • Passing fixture produces an all-green release-gate report and zero exit when --fail-on-breach=true.
  • Failing fixture produces gate failures and exit code 2 when --fail-on-breach=true.
  • Failing fixture with --auto-rollback=true executes endpoint canary strategy rollback flow and emits rollback execution evidence in report output.