Date: 2026-02-22
Checklist Items: REL-001, REL-002, REL-003, REL-004
Tickets: ISIS-0175, ISIS-0176, ISIS-0177, ISIS-0178
Purpose#
Provide deterministic controls for queue cutover:
- global queue dispatch toggle (
REL-001) - canary routing to a limited project set + traffic percentage (
REL-002) - canary SLO/cost verification gate (
REL-003) - ramp automation + auto-rollback decisioning (
REL-004)
Flags#
- Environment variable:
ISIS_QUEUE_PRODUCER_ENABLED - Type: boolean (
true/false, plus1/0,yes/no,on/off) - Default:
true - Environment variable:
ISIS_QUEUE_PRODUCER_CANARY_ENABLED - Type: boolean (
true/false, plus1/0,yes/no,on/off) - Default:
false - Environment variable:
ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE - Type: integer
0-100 - Default:
0 - Environment variable:
ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS - Type: comma-separated project IDs
- Default: empty (all projects eligible for canary percentage sampling)
Behavior#
When ISIS_QUEUE_PRODUCER_ENABLED=true:
- generation API enqueues claimed outbox entries to BullMQ.
- startup enforces production queue env requirements (
ISIS_QUEUE_NAMEandISIS_QUEUE_REDIS_URLorREDIS_URL). - submit path performs immediate outbox claim + dispatch attempt.
- outbox relay timer runs and retries failed dispatches.
- queue lifecycle listener can synchronize worker ACK/NACK updates.
When ISIS_QUEUE_PRODUCER_ENABLED=false and canary is disabled:
- submit persists job + outbox entry but returns
pending(dispatch deferred). - outbox relay dispatch loop is paused (no enqueue attempts).
- immediate submit-time enqueue is skipped.
- queue drain admin action is blocked.
- startup skips production queue env requirement enforcement.
- queue health/readiness reports queue as healthy-idle so service can remain available during staged rollout.
When ISIS_QUEUE_PRODUCER_ENABLED=false and canary is enabled:
- queue dispatch is active only for rollout-eligible traffic.
- rollout eligibility is evaluated per submit:
- optional project allowlist gate (
ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS) - deterministic bucket
< ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE
- optional project allowlist gate (
- submit returns:
queuedfor sampled-in canary jobspendingfor sampled-out / non-allowlisted jobs
- outbox relay claims only
metadata.queueDispatch.rolloutEligible=trueentries so non-canary jobs remain deferred. - production startup enforces queue env requirements when canary dispatch is active.
Rollout Guidance#
- Deploy with
ISIS_QUEUE_PRODUCER_ENABLED=falseto validate non-dispatch surfaces safely. - Enable canary for a restricted cohort:
ISIS_QUEUE_PRODUCER_CANARY_ENABLED=trueISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<small value>ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS=<project-a,project-b>
- Verify health/readiness and baseline application checks.
- Confirm sampled projects receive queue dispatch while control traffic remains pending.
- Run canary verification gate and capture report artifact:
scripts/isis/verify_canary_slo_cost.sh- reference:
docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
- Use ramp controller for promote/rollback decision:
scripts/isis/advance_queue_canary_rollout.sh- reference:
docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md
- Expand percentage/project scope during canary only if gate checks pass.
- Conduct pre-cutover game-day harness and capture report:
scripts/isis/run_release_gameday.sh- references:
docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md,docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
- Enable full queue dispatch with
ISIS_QUEUE_PRODUCER_ENABLED=true. - Run post-cutover validation and sign-off harness:
scripts/isis/run_post_cutover_validation.sh- references:
docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md,docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md
- If rollback is needed, set
ISIS_QUEUE_PRODUCER_ENABLED=falseandISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0, then restart.
Validation Signals#
/readyremains healthy with queue dependency reportingokin disabled mode.POST /api/v1/jobsreturns:pendingfor control trafficqueuedfor sampled canary traffic.
- Outbox rows stay
pendingwhile disabled and transition toenqueuedonce dispatch is re-enabled. - In canary-only mode, relay processing advances only rollout-eligible outbox rows.
POST /api/v1/jobs/admin/queue/drainfails closed while disabled.GET /api/v1/jobs/feature-flagsexposes queue dispatch and canary diagnostics under.queue.
Related Artifacts#
apps/isis/generation-api/src/services/job.service.tsapps/isis/generation-api/src/queue/producer.tsapps/isis/generation-api/src/app.tsdocs/domains/isis/api/README.mddocs/domains/isis/developer-guide.mddocs/domains/isis/README.mddocs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.mddocs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.mddocs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.mddocs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.mddocs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.mddocs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.mdscripts/isis/advance_queue_canary_rollout.shscripts/isis/run_release_gameday.shscripts/isis/run_post_cutover_validation.shscripts/isis/verify_canary_slo_cost.sh