Eve library

OPS-004 Certification: RunPod Budget Guardrails and Automatic Throttling

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: OPS-004
Ticket: ISIS-0239

Scope#

Operational cost-protection coverage for RunPod-backed generation paths:

  1. Threshold-based budget guardrails (soft/hard spend).
  2. Automatic throttling on generation write-path actions.
  3. Operator visibility of guardrail state in RunPod operations payload.
  4. Deterministic verification procedure and runbook controls.

Evidence Matrix#

Requirement Evidence Status
Budget guardrail logic exists in production write path apps/isis/generation-api/src/middleware/runpod-budget-guardrail.ts implements threshold evaluation, cached cost snapshot ingestion, soft-limit throttle windows, hard-limit blocking, and deterministic headers. Pass
Submit/batch/retry endpoints enforce guardrail automatically apps/isis/generation-api/src/routes/jobs.ts evaluates guardrail before abuse-control checks for POST /jobs, POST /jobs/batch, and POST /jobs/:id/retry, returning 429 + RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED when blocked. Pass
Guardrail status is operationally visible Same route file exposes guardrail state in GET /api/v1/jobs/runpod/operations as runpod.guardrail, including mode, thresholds, throttle window usage, and cost-source metadata. Pass
Runtime enforcement has automated tests apps/isis/generation-api/src/routes/jobs.spec.ts includes hard-limit block and soft-limit throttle exhaustion tests; apps/isis/generation-api/src/middleware/runpod-budget-guardrail.spec.ts covers core decision logic. Pass
Runbook exists with deployment and rollback controls docs/domains/isis/runbooks/runpod-budget-guardrails.md defines env controls, rollout sequence, verification commands, alert expectations, and rollback procedure. Pass
Verification automation exists scripts/isis/verify_runpod_budget_guardrails.sh validates operations payload guardrail state and synthetic submit behavior (allowed vs correctly throttled) with machine-readable JSON report output. Pass

Reproduction Commands#

bash
pnpm --filter @isis/generation-api exec vitest run src/middleware/runpod-budget-guardrail.spec.ts src/routes/jobs.spec.ts
bash -n scripts/isis/verify_runpod_budget_guardrails.sh

STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh