Eve library

ISIS Canary Ramp and Auto-Rollback (REL-004)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: REL-004
Ticket: ISIS-0178

Purpose#

Automate percentage ramp decisions for queue canary rollout with rollback-safe controls tied to objective SLO and cost thresholds.

This flow builds on:

  • REL-001: queue producer global dispatch toggle
  • REL-002: queue canary project/percentage controls
  • REL-003: canary SLO/cost verifier

Controller Script#

  • Script: scripts/isis/advance_queue_canary_rollout.sh
  • Verification dependency: scripts/isis/verify_canary_slo_cost.sh
  • Output artifacts:
    • verification report JSON
    • rollout decision JSON (promote, hold, rollback, steady)

The controller itself does not mutate deployment configuration unless explicit callbacks are provided.

Default Ramp Sequence#

Default canary progression:

  • 5,10,25,50,100

Override via RAMP_SEQUENCE.

Usage#

Decision-Only (No Mutation)#

bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh

Strict Auto-Rollback Mode#

bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh

If verification fails and auto-rollback is enabled:

  • decision action becomes rollback
  • target percentage is 0
  • script exits non-zero when FAIL_ON_ROLLBACK=true

With Apply/Rollback Callbacks#

Callbacks run in shell with context env vars (ACTION, CURRENT_PERCENTAGE, TARGET_PERCENTAGE, VERIFICATION_REPORT_FILE, DECISION_FILE, and FAILED_CHECKS on rollback).

bash
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
APPLY_NEXT_PERCENTAGE_CALLBACK='echo "apply ${TARGET_PERCENTAGE}"' \
ROLLBACK_CALLBACK='echo "rollback to ${TARGET_PERCENTAGE}"' \
scripts/isis/advance_queue_canary_rollout.sh

Decision Model#

promote#

  • Triggered when verification passes and a higher ramp percentage exists.
  • Target set to next percentage in sequence.

steady#

  • Triggered when verification passes and canary is already at max percentage.

rollback#

  • Triggered when verification fails and AUTO_ROLLBACK_ENABLED=true.
  • Target forced to 0.

hold#

  • Triggered when verification fails and auto-rollback is disabled.

Suggested Deployment Integration#

  1. Run verify_canary_slo_cost.sh (or let controller invoke it).
  2. Run advance_queue_canary_rollout.sh with current percentage.
  3. On promote, update deployment env:
    • ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<targetPercentage>
    • restart generation API
  4. On rollback, set:
    • ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0
    • optionally ISIS_QUEUE_PRODUCER_CANARY_ENABLED=false
    • restart generation API
  5. Attach verification + decision JSON to release evidence package.

Core Configuration#

  • CURRENT_PERCENTAGE (required): current queue canary percentage
  • RAMP_SEQUENCE (optional): comma-separated integer percentages
  • AUTO_ROLLBACK_ENABLED (default true)
  • FAIL_ON_ROLLBACK (default true)
  • APPLY_NEXT_PERCENTAGE_CALLBACK (optional)
  • ROLLBACK_CALLBACK (optional)
  • VERIFY_SCRIPT (optional path override)
  • REPORT_DIR, VERIFICATION_REPORT_FILE, DECISION_FILE (artifact paths)

All REL-003 verifier threshold overrides are supported via inherited env vars (e.g., QUEUE_DELAY_SLO_TARGET, COST_BURN_REGRESSION_MAX, MIN_CANARY_TERMINAL_JOBS, SUBMIT_LATENCY_SLI).

Traceability#

  • REL-004: Gradual canary percentage increase with auto-rollback thresholds
  • REL-005: game-day harness should consume ramp decision artifacts
  • Related docs:
    • docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
    • docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md
    • docs/domains/isis/remediation/ISIS_QUEUE_PRODUCER_ROLLOUT_FLAG.md
    • docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md