Eve library

ISIS v2 WS14-004: Queue Health Dashboards with Saturation and Starvation Indicators

Guide

Revision 1

On this page

Scope#

WS14-004 adds queue-health dashboards that explicitly surface saturation and starvation indicators derived from queue depth, pressure, stuck-job age, and throughput signals.

Research and Existing Implementation Alignment#

Before implementation, existing queue telemetry paths were analyzed:

  • queue producer health primitives (waiting, active, delayed, failed):
    • apps/isis/generation-api/src/queue/producer.ts
  • queue statistical telemetry (queueDepth, stuckJobs, retries, DLQ size):
    • apps/isis/generation-api/src/services/job.service.ts
    • apps/isis/generation-api/src/types.ts
  • provider operations aggregation route:
    • apps/isis/generation-api/src/routes/jobs.ts
  • provider operations dashboard rendering:
    • apps/isis/web/src/pages/ProvidersPage.tsx
  • SDK provider-operations contract/date parsing:
    • libs/isis/client/src/types.ts
    • libs/isis/client/src/resources/generation.ts

This baseline had queue stats and health snapshots, but no derived saturation and starvation indicators in API/dashboard contracts.

Deliverables#

  1. Added queue indicator model and scoring in generation API route logic:
  • apps/isis/generation-api/src/routes/jobs.ts

Changes:

  • introduced buildQueueHealthIndicators() with deterministic scoring,
  • added explicit saturation indicators (queue pressure ratio, stuck jobs, delayed jobs, queue depth),
  • added starvation indicators (zero throughput under depth, stuck-age threshold breaches, wait inflation, DLQ pressure),
  • published indicators in /api/v1/jobs/providers/operations response under queue.indicators.
  1. Added API route contract regression coverage:
  • apps/isis/generation-api/src/routes/jobs.spec.ts

New assertions verify:

  • saturation/starvation indicator presence,
  • critical-level classification under stressed queue conditions,
  • reason strings for starvation/saturation trigger attribution.
  1. Extended SDK contract + date parsing for queue indicators:
  • libs/isis/client/src/types.ts
  • libs/isis/client/src/resources/generation.ts

Added queue indicator interfaces and parsing for generatedAt.

  1. Added queue indicator dashboard rendering in Providers page:
  • apps/isis/web/src/pages/ProvidersPage.tsx

Dashboard now displays:

  • saturation level/score,
  • starvation level/score,
  • pressure ratio and stuck-age metrics,
  • reason traces for saturation and starvation diagnostics.
  1. Extended Providers page rendering expectations:
  • apps/isis/web/src/pages/ProvidersPage.spec.tsx

Added indicator fixture data and dashboard text assertions.

Verification Commands#

bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/ProvidersPage.spec.tsx --reporter=dot
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • @isis/generation-api route suite passes with indicator assertions.
  • @isis/web Providers page vitest remains blocked by known @isis/client package-entry resolution issue.
  • @isis/web typecheck remains at known baseline issues outside WS14-004 scope (libs/isis/client/src/resources/workflows.ts typing and existing rootDir configuration constraints).