# ISIS Queue Producer Rollout and Canary Flags (REL-001, REL-002, REL-003, REL-004)

Date: 2026-02-22  
Checklist Items: `REL-001`, `REL-002`, `REL-003`, `REL-004`  
Tickets: `ISIS-0175`, `ISIS-0176`, `ISIS-0177`, `ISIS-0178`

## Purpose

Provide deterministic controls for queue cutover:

- global queue dispatch toggle (`REL-001`)
- canary routing to a limited project set + traffic percentage (`REL-002`)
- canary SLO/cost verification gate (`REL-003`)
- ramp automation + auto-rollback decisioning (`REL-004`)

## Flags

- Environment variable: `ISIS_QUEUE_PRODUCER_ENABLED`
- Type: boolean (`true`/`false`, plus `1/0`, `yes/no`, `on/off`)
- Default: `true`
- Environment variable: `ISIS_QUEUE_PRODUCER_CANARY_ENABLED`
- Type: boolean (`true`/`false`, plus `1/0`, `yes/no`, `on/off`)
- Default: `false`
- Environment variable: `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE`
- Type: integer `0-100`
- Default: `0`
- Environment variable: `ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS`
- Type: comma-separated project IDs
- Default: empty (all projects eligible for canary percentage sampling)

## Behavior

When `ISIS_QUEUE_PRODUCER_ENABLED=true`:

- generation API enqueues claimed outbox entries to BullMQ.
- startup enforces production queue env requirements (`ISIS_QUEUE_NAME` and
  `ISIS_QUEUE_REDIS_URL` or `REDIS_URL`).
- submit path performs immediate outbox claim + dispatch attempt.
- outbox relay timer runs and retries failed dispatches.
- queue lifecycle listener can synchronize worker ACK/NACK updates.

When `ISIS_QUEUE_PRODUCER_ENABLED=false` and canary is disabled:

- submit persists job + outbox entry but returns `pending` (dispatch deferred).
- outbox relay dispatch loop is paused (no enqueue attempts).
- immediate submit-time enqueue is skipped.
- queue drain admin action is blocked.
- startup skips production queue env requirement enforcement.
- queue health/readiness reports queue as healthy-idle so service can remain
  available during staged rollout.

When `ISIS_QUEUE_PRODUCER_ENABLED=false` and canary is enabled:

- queue dispatch is active only for rollout-eligible traffic.
- rollout eligibility is evaluated per submit:
  - optional project allowlist gate (`ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS`)
  - deterministic bucket `< ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE`
- submit returns:
  - `queued` for sampled-in canary jobs
  - `pending` for sampled-out / non-allowlisted jobs
- outbox relay claims only `metadata.queueDispatch.rolloutEligible=true` entries
  so non-canary jobs remain deferred.
- production startup enforces queue env requirements when canary dispatch is
  active.

## Rollout Guidance

1. Deploy with `ISIS_QUEUE_PRODUCER_ENABLED=false` to validate non-dispatch
   surfaces safely.
2. Enable canary for a restricted cohort:
   - `ISIS_QUEUE_PRODUCER_CANARY_ENABLED=true`
   - `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=<small value>`
   - `ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS=<project-a,project-b>`
3. Verify health/readiness and baseline application checks.
4. Confirm sampled projects receive queue dispatch while control traffic remains
   pending.
5. Run canary verification gate and capture report artifact:
   - `scripts/isis/verify_canary_slo_cost.sh`
   - reference:
     `docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md`
6. Use ramp controller for promote/rollback decision:
   - `scripts/isis/advance_queue_canary_rollout.sh`
   - reference:
     `docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md`
7. Expand percentage/project scope during canary only if gate checks pass.
8. Conduct pre-cutover game-day harness and capture report:
   - `scripts/isis/run_release_gameday.sh`
   - references: `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`,
     `docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md`
9. Enable full queue dispatch with `ISIS_QUEUE_PRODUCER_ENABLED=true`.
10. Run post-cutover validation and sign-off harness:
    - `scripts/isis/run_post_cutover_validation.sh`
    - references:
      `docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md`,
      `docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md`
11. If rollback is needed, set `ISIS_QUEUE_PRODUCER_ENABLED=false` and
    `ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=0`, then restart.

## Validation Signals

- `/ready` remains healthy with queue dependency reporting `ok` in disabled
  mode.
- `POST /api/v1/jobs` returns:
  - `pending` for control traffic
  - `queued` for sampled canary traffic.
- Outbox rows stay `pending` while disabled and transition to `enqueued` once
  dispatch is re-enabled.
- In canary-only mode, relay processing advances only rollout-eligible outbox
  rows.
- `POST /api/v1/jobs/admin/queue/drain` fails closed while disabled.
- `GET /api/v1/jobs/feature-flags` exposes queue dispatch and canary diagnostics
  under `.queue`.

## Related Artifacts

- `apps/isis/generation-api/src/services/job.service.ts`
- `apps/isis/generation-api/src/queue/producer.ts`
- `apps/isis/generation-api/src/app.ts`
- `docs/domains/isis/api/README.md`
- `docs/domains/isis/developer-guide.md`
- `docs/domains/isis/README.md`
- `docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md`
- `docs/domains/isis/remediation/ISIS_CANARY_ROLLOUT_AUTOROLLBACK.md`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_CUTOVER_PLAN.md`
- `docs/domains/isis/remediation/ISIS_GAMEDAY_REPORT_TEMPLATE.md`
- `docs/domains/isis/remediation/ISIS_POST_CUTOVER_VALIDATION_CHECKLIST.md`
- `docs/domains/isis/remediation/ISIS_CUTOVER_SIGNOFF_TEMPLATE.md`
- `scripts/isis/advance_queue_canary_rollout.sh`
- `scripts/isis/run_release_gameday.sh`
- `scripts/isis/run_post_cutover_validation.sh`
- `scripts/isis/verify_canary_slo_cost.sh`
