Scope#
Implemented per-endpoint RunPod SLO telemetry for latency, success rate, timeout rate, and queue wait, and exposed these SLOs across API, SDK, and web operations surfaces.
Implemented Artifacts#
- RunPod adapter SLO contracts and runtime aggregation:
libs/isis/ai-providers/src/providers/comfy/types.tslibs/isis/ai-providers/src/providers/comfy/index.tslibs/isis/ai-providers/src/providers/comfy/runpod-adapter.tslibs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts
- Generation API SLO surface:
apps/isis/generation-api/src/services/comfyui.service.tsapps/isis/generation-api/src/routes/jobs.tsapps/isis/generation-api/src/routes/jobs.spec.ts
- SDK and web UI exposure:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.tslibs/isis/client/src/index.tsapps/isis/web/src/pages/RunPodPage.tsxapps/isis/web/src/pages/RunPodPage.spec.tsx
SLO Model#
- Per-endpoint rolling samples are captured from completed/failed/timeout jobs.
- SLO objective defaults:
- latency p95:
<= 180000ms - success rate:
>= 99% - timeout rate:
<= 1% - queue wait p95:
<= 30000ms - minimum sample size before breach evaluation:
5
- latency p95:
/api/v1/jobs/runpod/operationsnow includes endpoint SLO snapshots with:- objective thresholds
- endpoint sample sizes
- computed p95/percentage metrics
- breach booleans per SLO dimension
- endpoint topology enrichment (
family,channel)
Verification#
bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
Outcome#
WS09-006 is now closed with first-class per-endpoint SLO observability across RunPod lanes, enabling objective breach detection for latency, reliability, and queueing behavior in both API and operator UI.