Eve library

ISIS v2 WS09-007: RunPod Anomaly Detection

Guide

Revision 1

On this page

Scope#

Implemented RunPod anomaly detection for cost spikes, timeout storms, and model cache thrashing, and exposed anomaly telemetry through API, SDK parsing, and web operations UI.

Implemented Artifacts#

  • RunPod adapter anomaly contracts and detection runtime:
    • libs/isis/ai-providers/src/providers/comfy/types.ts
    • libs/isis/ai-providers/src/providers/comfy/index.ts
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts
  • Generation API anomaly surface:
    • apps/isis/generation-api/src/services/comfyui.service.ts
    • apps/isis/generation-api/src/routes/jobs.ts
    • apps/isis/generation-api/src/routes/jobs.spec.ts
  • SDK + web UI exposure:
    • libs/isis/client/src/types.ts
    • libs/isis/client/src/resources/generation.ts
    • libs/isis/client/src/index.ts
    • apps/isis/web/src/pages/RunPodPage.tsx
    • apps/isis/web/src/pages/RunPodPage.spec.tsx

Detection Model#

  • Cost spike detection:
    • evaluates recent completed-job USD series in a sliding window
    • computes baseline mean/std-dev and z-score for latest sample
    • requires z-score threshold and minimum USD delta
  • Timeout storm detection:
    • evaluates per-endpoint timeout rate over recent endpoint outcome samples
    • requires both minimum timeout count and timeout-rate threshold
  • Model cache thrashing detection:
    • evaluates per-endpoint model-signature churn (distinct signatures / samples)
    • combines churn thresholds with queue-wait p95 and model/cache-related error signals
  • /api/v1/jobs/runpod/operations now includes runpod.anomalies with:
    • objectives/thresholds
    • cost-spike snapshot
    • timeout-storm endpoint entries
    • model-cache-thrashing endpoint entries
    • active anomaly alerts

Verification#

bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts src/services/comfyui.service.spec.ts src/middleware/runpod-budget-guardrail.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx src/pages/JobSubmissionStudio.spec.tsx --reporter=dot

Outcome#

WS09-007 is now closed with first-class anomaly detection and alert surfacing for RunPod cost spikes, timeout storms, and model cache thrashing across backend and operator UI telemetry planes.