# ISIS v2 WS14-004: Queue Health Dashboards with Saturation and Starvation Indicators

## Scope

`WS14-004` adds queue-health dashboards that explicitly surface saturation and
starvation indicators derived from queue depth, pressure, stuck-job age, and
throughput signals.

## Research and Existing Implementation Alignment

Before implementation, existing queue telemetry paths were analyzed:

- queue producer health primitives (`waiting`, `active`, `delayed`, `failed`):
  - `apps/isis/generation-api/src/queue/producer.ts`
- queue statistical telemetry (`queueDepth`, `stuckJobs`, retries, DLQ size):
  - `apps/isis/generation-api/src/services/job.service.ts`
  - `apps/isis/generation-api/src/types.ts`
- provider operations aggregation route:
  - `apps/isis/generation-api/src/routes/jobs.ts`
- provider operations dashboard rendering:
  - `apps/isis/web/src/pages/ProvidersPage.tsx`
- SDK provider-operations contract/date parsing:
  - `libs/isis/client/src/types.ts`
  - `libs/isis/client/src/resources/generation.ts`

This baseline had queue stats and health snapshots, but no derived saturation
and starvation indicators in API/dashboard contracts.

## Deliverables

1. Added queue indicator model and scoring in generation API route logic:

- `apps/isis/generation-api/src/routes/jobs.ts`

Changes:

- introduced `buildQueueHealthIndicators()` with deterministic scoring,
- added explicit saturation indicators (queue pressure ratio, stuck jobs,
  delayed jobs, queue depth),
- added starvation indicators (zero throughput under depth, stuck-age threshold
  breaches, wait inflation, DLQ pressure),
- published indicators in `/api/v1/jobs/providers/operations` response under
  `queue.indicators`.

2. Added API route contract regression coverage:

- `apps/isis/generation-api/src/routes/jobs.spec.ts`

New assertions verify:

- saturation/starvation indicator presence,
- critical-level classification under stressed queue conditions,
- reason strings for starvation/saturation trigger attribution.

3. Extended SDK contract + date parsing for queue indicators:

- `libs/isis/client/src/types.ts`
- `libs/isis/client/src/resources/generation.ts`

Added queue indicator interfaces and parsing for `generatedAt`.

4. Added queue indicator dashboard rendering in Providers page:

- `apps/isis/web/src/pages/ProvidersPage.tsx`

Dashboard now displays:

- saturation level/score,
- starvation level/score,
- pressure ratio and stuck-age metrics,
- reason traces for saturation and starvation diagnostics.

5. Extended Providers page rendering expectations:

- `apps/isis/web/src/pages/ProvidersPage.spec.tsx`

Added indicator fixture data and dashboard text assertions.

## Verification Commands

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/ProvidersPage.spec.tsx --reporter=dot
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

## Verification Notes

- `@isis/generation-api` route suite passes with indicator assertions.
- `@isis/web` Providers page vitest remains blocked by known `@isis/client`
  package-entry resolution issue.
- `@isis/web` typecheck remains at known baseline issues outside `WS14-004`
  scope (`libs/isis/client/src/resources/workflows.ts` typing and existing
  `rootDir` configuration constraints).
