Date: 2026-02-22
Checklist Item: REL-004
Ticket: ISIS-0178
Purpose#
Automate percentage ramp decisions for queue canary rollout with rollback-safe controls tied to objective SLO and cost thresholds.
This flow builds on:
REL-001: queue producer global dispatch toggleREL-002: queue canary project/percentage controlsREL-003: canary SLO/cost verifier
Controller Script#
- Script:
scripts/isis/advance_queue_canary_rollout.sh - Verification dependency:
scripts/isis/verify_canary_slo_cost.sh - Output artifacts:
- verification report JSON
- rollout decision JSON (
promote,hold,rollback,steady)
The controller itself does not mutate deployment configuration unless explicit callbacks are provided.
Default Ramp Sequence#
Default canary progression:
5,10,25,50,100
Override via RAMP_SEQUENCE.
Usage#
Decision-Only (No Mutation)#
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
scripts/isis/advance_queue_canary_rollout.sh
Strict Auto-Rollback Mode#
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=true \
scripts/isis/advance_queue_canary_rollout.sh
If verification fails and auto-rollback is enabled:
- decision action becomes
rollback - target percentage is
0 - script exits non-zero when
FAIL_ON_ROLLBACK=true
With Apply/Rollback Callbacks#
Callbacks run in shell with context env vars (ACTION, CURRENT_PERCENTAGE,
TARGET_PERCENTAGE, VERIFICATION_REPORT_FILE, DECISION_FILE, and
FAILED_CHECKS on rollback).
ISIS_DATABASE_URL=postgres://... \
CURRENT_PERCENTAGE=10 \
APPLY_NEXT_PERCENTAGE_CALLBACK='echo "apply ${TARGET_PERCENTAGE}"' \
ROLLBACK_CALLBACK='echo "rollback to ${TARGET_PERCENTAGE}"' \
scripts/isis/advance_queue_canary_rollout.sh
Decision Model#
promote#
- Triggered when verification passes and a higher ramp percentage exists.
- Target set to next percentage in sequence.
steady#
- Triggered when verification passes and canary is already at max percentage.
rollback#
- Triggered when verification fails and
AUTO_ROLLBACK_ENABLED=true. - Target forced to
0.
hold#
- Triggered when verification fails and auto-rollback is disabled.
Suggested Deployment Integration#
- Run
verify_canary_slo_cost.sh(or let controller invoke it). - Run
advance_queue_canary_rollout.shwith current percentage. - On
promote, update deployment env:ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<targetPercentage>- restart generation API
- On
rollback, set:ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0- optionally
ISIS_QUEUE_PRODUCER_CANARY_ENABLED=false - restart generation API
- Attach verification + decision JSON to release evidence package.
Core Configuration#
CURRENT_PERCENTAGE(required): current queue canary percentageRAMP_SEQUENCE(optional): comma-separated integer percentagesAUTO_ROLLBACK_ENABLED(defaulttrue)FAIL_ON_ROLLBACK(defaulttrue)APPLY_NEXT_PERCENTAGE_CALLBACK(optional)ROLLBACK_CALLBACK(optional)VERIFY_SCRIPT(optional path override)REPORT_DIR,VERIFICATION_REPORT_FILE,DECISION_FILE(artifact paths)
All REL-003 verifier threshold overrides are supported via inherited env vars
(e.g., QUEUE_DELAY_SLO_TARGET, COST_BURN_REGRESSION_MAX,
MIN_CANARY_TERMINAL_JOBS, SUBMIT_LATENCY_SLI).
Traceability#
REL-004: Gradual canary percentage increase with auto-rollback thresholdsREL-005: game-day harness should consume ramp decision artifacts- Related docs:
docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.mddocs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.mddocs/domains/isis/remediation/ISIS_QUEUE_PRODUCER_ROLLOUT_FLAG.mddocs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md