# ISIS v2 WS09-006: RunPod Per-Endpoint SLOs

## Scope

Implemented per-endpoint RunPod SLO telemetry for latency, success rate, timeout
rate, and queue wait, and exposed these SLOs across API, SDK, and web operations
surfaces.

## Implemented Artifacts

- RunPod adapter SLO contracts and runtime aggregation:
  - `libs/isis/ai-providers/src/providers/comfy/types.ts`
  - `libs/isis/ai-providers/src/providers/comfy/index.ts`
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts`
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts`
- Generation API SLO surface:
  - `apps/isis/generation-api/src/services/comfyui.service.ts`
  - `apps/isis/generation-api/src/routes/jobs.ts`
  - `apps/isis/generation-api/src/routes/jobs.spec.ts`
- SDK and web UI exposure:
  - `libs/isis/client/src/types.ts`
  - `libs/isis/client/src/resources/generation.ts`
  - `libs/isis/client/src/index.ts`
  - `apps/isis/web/src/pages/RunPodPage.tsx`
  - `apps/isis/web/src/pages/RunPodPage.spec.tsx`

## SLO Model

- Per-endpoint rolling samples are captured from completed/failed/timeout jobs.
- SLO objective defaults:
  - latency p95: `<= 180000ms`
  - success rate: `>= 99%`
  - timeout rate: `<= 1%`
  - queue wait p95: `<= 30000ms`
  - minimum sample size before breach evaluation: `5`
- `/api/v1/jobs/runpod/operations` now includes endpoint SLO snapshots with:
  - objective thresholds
  - endpoint sample sizes
  - computed p95/percentage metrics
  - breach booleans per SLO dimension
  - endpoint topology enrichment (`family`, `channel`)

## Verification

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

## Outcome

WS09-006 is now closed with first-class per-endpoint SLO observability across
RunPod lanes, enabling objective breach detection for latency, reliability, and
queueing behavior in both API and operator UI.
