Scope#
WS14-004 adds queue-health dashboards that explicitly surface saturation and
starvation indicators derived from queue depth, pressure, stuck-job age, and
throughput signals.
Research and Existing Implementation Alignment#
Before implementation, existing queue telemetry paths were analyzed:
- queue producer health primitives (
waiting,active,delayed,failed):apps/isis/generation-api/src/queue/producer.ts
- queue statistical telemetry (
queueDepth,stuckJobs, retries, DLQ size):apps/isis/generation-api/src/services/job.service.tsapps/isis/generation-api/src/types.ts
- provider operations aggregation route:
apps/isis/generation-api/src/routes/jobs.ts
- provider operations dashboard rendering:
apps/isis/web/src/pages/ProvidersPage.tsx
- SDK provider-operations contract/date parsing:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.ts
This baseline had queue stats and health snapshots, but no derived saturation and starvation indicators in API/dashboard contracts.
Deliverables#
- Added queue indicator model and scoring in generation API route logic:
apps/isis/generation-api/src/routes/jobs.ts
Changes:
- introduced
buildQueueHealthIndicators()with deterministic scoring, - added explicit saturation indicators (queue pressure ratio, stuck jobs, delayed jobs, queue depth),
- added starvation indicators (zero throughput under depth, stuck-age threshold breaches, wait inflation, DLQ pressure),
- published indicators in
/api/v1/jobs/providers/operationsresponse underqueue.indicators.
- Added API route contract regression coverage:
apps/isis/generation-api/src/routes/jobs.spec.ts
New assertions verify:
- saturation/starvation indicator presence,
- critical-level classification under stressed queue conditions,
- reason strings for starvation/saturation trigger attribution.
- Extended SDK contract + date parsing for queue indicators:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.ts
Added queue indicator interfaces and parsing for generatedAt.
- Added queue indicator dashboard rendering in Providers page:
apps/isis/web/src/pages/ProvidersPage.tsx
Dashboard now displays:
- saturation level/score,
- starvation level/score,
- pressure ratio and stuck-age metrics,
- reason traces for saturation and starvation diagnostics.
- Extended Providers page rendering expectations:
apps/isis/web/src/pages/ProvidersPage.spec.tsx
Added indicator fixture data and dashboard text assertions.
Verification Commands#
bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/ProvidersPage.spec.tsx --reporter=dot
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
@isis/generation-apiroute suite passes with indicator assertions.@isis/webProviders page vitest remains blocked by known@isis/clientpackage-entry resolution issue.@isis/webtypecheck remains at known baseline issues outsideWS14-004scope (libs/isis/client/src/resources/workflows.tstyping and existingrootDirconfiguration constraints).