Scope#
Implemented RunPod anomaly detection for cost spikes, timeout storms, and model cache thrashing, and exposed anomaly telemetry through API, SDK parsing, and web operations UI.
Implemented Artifacts#
- RunPod adapter anomaly contracts and detection runtime:
libs/isis/ai-providers/src/providers/comfy/types.tslibs/isis/ai-providers/src/providers/comfy/index.tslibs/isis/ai-providers/src/providers/comfy/runpod-adapter.tslibs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts
- Generation API anomaly surface:
apps/isis/generation-api/src/services/comfyui.service.tsapps/isis/generation-api/src/routes/jobs.tsapps/isis/generation-api/src/routes/jobs.spec.ts
- SDK + web UI exposure:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.tslibs/isis/client/src/index.tsapps/isis/web/src/pages/RunPodPage.tsxapps/isis/web/src/pages/RunPodPage.spec.tsx
Detection Model#
- Cost spike detection:
- evaluates recent completed-job USD series in a sliding window
- computes baseline mean/std-dev and z-score for latest sample
- requires z-score threshold and minimum USD delta
- Timeout storm detection:
- evaluates per-endpoint timeout rate over recent endpoint outcome samples
- requires both minimum timeout count and timeout-rate threshold
- Model cache thrashing detection:
- evaluates per-endpoint model-signature churn (distinct signatures / samples)
- combines churn thresholds with queue-wait p95 and model/cache-related error signals
/api/v1/jobs/runpod/operationsnow includesrunpod.anomalieswith:- objectives/thresholds
- cost-spike snapshot
- timeout-storm endpoint entries
- model-cache-thrashing endpoint entries
- active anomaly alerts
Verification#
bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot
Outcome#
WS09-007 is now closed with first-class anomaly detection and alert surfacing for RunPod cost spikes, timeout storms, and model cache thrashing across backend and operator UI telemetry planes.