# ISIS v2 WS09-007: RunPod Anomaly Detection

## Scope

Implemented RunPod anomaly detection for cost spikes, timeout storms, and model
cache thrashing, and exposed anomaly telemetry through API, SDK parsing, and web
operations UI.

## Implemented Artifacts

- RunPod adapter anomaly contracts and detection runtime:
  - `libs/isis/ai-providers/src/providers/comfy/types.ts`
  - `libs/isis/ai-providers/src/providers/comfy/index.ts`
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts`
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts`
- Generation API anomaly surface:
  - `apps/isis/generation-api/src/services/comfyui.service.ts`
  - `apps/isis/generation-api/src/routes/jobs.ts`
  - `apps/isis/generation-api/src/routes/jobs.spec.ts`
- SDK + web UI exposure:
  - `libs/isis/client/src/types.ts`
  - `libs/isis/client/src/resources/generation.ts`
  - `libs/isis/client/src/index.ts`
  - `apps/isis/web/src/pages/RunPodPage.tsx`
  - `apps/isis/web/src/pages/RunPodPage.spec.tsx`

## Detection Model

- Cost spike detection:
  - evaluates recent completed-job USD series in a sliding window
  - computes baseline mean/std-dev and z-score for latest sample
  - requires z-score threshold and minimum USD delta
- Timeout storm detection:
  - evaluates per-endpoint timeout rate over recent endpoint outcome samples
  - requires both minimum timeout count and timeout-rate threshold
- Model cache thrashing detection:
  - evaluates per-endpoint model-signature churn (distinct signatures / samples)
  - combines churn thresholds with queue-wait p95 and model/cache-related error
    signals
- `/api/v1/jobs/runpod/operations` now includes `runpod.anomalies` with:
  - objectives/thresholds
  - cost-spike snapshot
  - timeout-storm endpoint entries
  - model-cache-thrashing endpoint entries
  - active anomaly alerts

## Verification

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
```

## Outcome

WS09-007 is now closed with first-class anomaly detection and alert surfacing
for RunPod cost spikes, timeout storms, and model cache thrashing across backend
and operator UI telemetry planes.
