Eve library

Isis RunPod Budget Guardrails and Automatic Throttling Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-22
Scope: Automatic cost guardrails for RunPod-backed generation traffic
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team

Purpose#

Define deterministic controls that automatically throttle generation submissions when RunPod spend crosses configured thresholds.

This runbook covers:

  1. Guardrail configuration for soft/hard spend thresholds.
  2. Automatic throttling behavior for submit, batch submit, and retry flows.
  3. Verification procedure for guardrail status and enforcement.

Runtime Behavior Summary#

When enabled, generation API evaluates RunPod monthly cost telemetry before accepting expensive write-path actions (POST /api/v1/jobs, POST /api/v1/jobs/batch, POST /api/v1/jobs/:id/retry):

  • below soft limit: request allowed.
  • at/above soft limit: request path enters throttled mode and uses a global throttle units window.
  • at/above hard limit: request is rejected with 429 and RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED.

Guardrail status is exposed in GET /api/v1/jobs/runpod/operations at runpod.guardrail.

Configuration#

Environment Variable Default Description
ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED false Enable automatic RunPod spend guardrails on write-path generation actions.
ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD 0 Monthly estimated spend threshold that activates throttle mode.
ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD 0 Monthly estimated spend threshold that blocks requests immediately.
ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW 120 Max weighted guardrail units allowed per throttle window in soft-limit mode.
ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS 60 Duration of throttle window in seconds.
ISIS_RUNPOD_BUDGET_COST_CACHE_SECONDS 60 Cache TTL for RunPod cost lookups used by guardrail decisions.
ISIS_RUNPOD_BUDGET_RETRY_AFTER_SECONDS 60 Retry hint returned for hard-limit block responses.
ISIS_RUNPOD_BUDGET_FAIL_CLOSED false If true, unavailable cost telemetry blocks requests instead of failing open.

Rollout Procedure#

  1. Choose thresholds from forecast and risk envelope.
  2. Start in staging with conservative limits:
bash
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=100
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=150
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=120
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=60
  1. Deploy generation API.
  2. Run guardrail verification script (below).
  3. Promote to production after staging verification is green.

Verification Procedure#

Run verification script:

bash
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh

The script validates:

  1. GET /api/v1/jobs/runpod/operations is reachable.
  2. runpod.guardrail.enabled=true is present in operations payload.
  3. Synthetic submit either succeeds under guardrail headers or is correctly throttled with RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED.

Report output is written to tmp/isis-runpod-budget-guardrails/.

Alert and Triage Expectations#

  • Guardrail mode should be monitored from runpod.guardrail.mode in operations payload and dashboards.
  • Sustained throttle/block mode requires incident triage for forecast correction, workload shaping, or provider routing adjustment.

Rollback#

Disable guardrail and redeploy generation API:

bash
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=false

Use rollback only with explicit risk acceptance when spend controls are managed through another active mechanism.

  • apps/isis/generation-api/src/middleware/runpod-budget-guardrail.ts
  • apps/isis/generation-api/src/routes/jobs.ts
  • scripts/isis/verify_runpod_budget_guardrails.sh
  • docs/domains/isis/runbooks/runpod-oncall.md