Eve library

ISIS Queue Producer Rollout and Canary Flags (REL-001, REL-002, REL-003, REL-004)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Items: REL-001, REL-002, REL-003, REL-004
Tickets: ISIS-0175, ISIS-0176, ISIS-0177, ISIS-0178

Purpose#

Provide deterministic controls for queue cutover:

  • global queue dispatch toggle (REL-001)
  • canary routing to a limited project set + traffic percentage (REL-002)
  • canary SLO/cost verification gate (REL-003)
  • ramp automation + auto-rollback decisioning (REL-004)

Flags#

  • Environment variable: ISIS_QUEUE_PRODUCER_ENABLED
  • Type: boolean (true/false, plus 1/0, yes/no, on/off)
  • Default: true
  • Environment variable: ISIS_QUEUE_PRODUCER_CANARY_ENABLED
  • Type: boolean (true/false, plus 1/0, yes/no, on/off)
  • Default: false
  • Environment variable: ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE
  • Type: integer 0-100
  • Default: 0
  • Environment variable: ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS
  • Type: comma-separated project IDs
  • Default: empty (all projects eligible for canary percentage sampling)

Behavior#

When ISIS_QUEUE_PRODUCER_ENABLED=true:

  • generation API enqueues claimed outbox entries to BullMQ.
  • startup enforces production queue env requirements (ISIS_QUEUE_NAME and ISIS_QUEUE_REDIS_URL or REDIS_URL).
  • submit path performs immediate outbox claim + dispatch attempt.
  • outbox relay timer runs and retries failed dispatches.
  • queue lifecycle listener can synchronize worker ACK/NACK updates.

When ISIS_QUEUE_PRODUCER_ENABLED=false and canary is disabled:

  • submit persists job + outbox entry but returns pending (dispatch deferred).
  • outbox relay dispatch loop is paused (no enqueue attempts).
  • immediate submit-time enqueue is skipped.
  • queue drain admin action is blocked.
  • startup skips production queue env requirement enforcement.
  • queue health/readiness reports queue as healthy-idle so service can remain available during staged rollout.

When ISIS_QUEUE_PRODUCER_ENABLED=false and canary is enabled:

  • queue dispatch is active only for rollout-eligible traffic.
  • rollout eligibility is evaluated per submit:
    • optional project allowlist gate (ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS)
    • deterministic bucket < ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE
  • submit returns:
    • queued for sampled-in canary jobs
    • pending for sampled-out / non-allowlisted jobs
  • outbox relay claims only metadata.queueDispatch.rolloutEligible=true entries so non-canary jobs remain deferred.
  • production startup enforces queue env requirements when canary dispatch is active.

Rollout Guidance#

  1. Deploy with ISIS_QUEUE_PRODUCER_ENABLED=false to validate non-dispatch surfaces safely.
  2. Enable canary for a restricted cohort:
    • ISIS_QUEUE_PRODUCER_CANARY_ENABLED=true
    • ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<small value>
    • ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS=<project-a,project-b>
  3. Verify health/readiness and baseline application checks.
  4. Confirm sampled projects receive queue dispatch while control traffic remains pending.
  5. Run canary verification gate and capture report artifact:
    • scripts/isis/verify_canary_slo_cost.sh
    • reference: docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
  6. Use ramp controller for promote/rollback decision:
    • scripts/isis/advance_queue_canary_rollout.sh
    • reference: docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md
  7. Expand percentage/project scope during canary only if gate checks pass.
  8. Conduct pre-cutover game-day harness and capture report:
    • scripts/isis/run_release_gameday.sh
    • references: docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md, docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
  9. Enable full queue dispatch with ISIS_QUEUE_PRODUCER_ENABLED=true.
  10. Run post-cutover validation and sign-off harness:
    • scripts/isis/run_post_cutover_validation.sh
    • references: docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md, docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md
  11. If rollback is needed, set ISIS_QUEUE_PRODUCER_ENABLED=false and ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0, then restart.

Validation Signals#

  • /ready remains healthy with queue dependency reporting ok in disabled mode.
  • POST /api/v1/jobs returns:
    • pending for control traffic
    • queued for sampled canary traffic.
  • Outbox rows stay pending while disabled and transition to enqueued once dispatch is re-enabled.
  • In canary-only mode, relay processing advances only rollout-eligible outbox rows.
  • POST /api/v1/jobs/admin/queue/drain fails closed while disabled.
  • GET /api/v1/jobs/feature-flags exposes queue dispatch and canary diagnostics under .queue.
  • apps/isis/generation-api/src/services/job.service.ts
  • apps/isis/generation-api/src/queue/producer.ts
  • apps/isis/generation-api/src/app.ts
  • docs/domains/isis/api/README.md
  • docs/domains/isis/developer-guide.md
  • docs/domains/isis/README.md
  • docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md
  • docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md
  • docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md
  • docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md
  • docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md
  • docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md
  • scripts/isis/advance_queue_canary_rollout.sh
  • scripts/isis/run_release_gameday.sh
  • scripts/isis/run_post_cutover_validation.sh
  • scripts/isis/verify_canary_slo_cost.sh