Last Updated: 2026-02-22
Scope: Automatic cost guardrails for RunPod-backed generation traffic
Primary Owners: @oshun/devops, @oshun/isis-team, @oshun/ml-team
Purpose#
Define deterministic controls that automatically throttle generation submissions when RunPod spend crosses configured thresholds.
This runbook covers:
- Guardrail configuration for soft/hard spend thresholds.
- Automatic throttling behavior for submit, batch submit, and retry flows.
- Verification procedure for guardrail status and enforcement.
Runtime Behavior Summary#
When enabled, generation API evaluates RunPod monthly cost telemetry before
accepting expensive write-path actions (POST /api/v1/jobs,
POST /api/v1/jobs/batch, POST /api/v1/jobs/:id/retry):
- below soft limit: request allowed.
- at/above soft limit: request path enters throttled mode and uses a global throttle units window.
- at/above hard limit: request is rejected with
429andRUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED.
Guardrail status is exposed in GET /api/v1/jobs/runpod/operations at
runpod.guardrail.
Configuration#
| Environment Variable | Default | Description |
|---|---|---|
ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED |
false |
Enable automatic RunPod spend guardrails on write-path generation actions. |
ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD |
0 |
Monthly estimated spend threshold that activates throttle mode. |
ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD |
0 |
Monthly estimated spend threshold that blocks requests immediately. |
ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW |
120 |
Max weighted guardrail units allowed per throttle window in soft-limit mode. |
ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS |
60 |
Duration of throttle window in seconds. |
ISIS_RUNPOD_BUDGET_COST_CACHE_SECONDS |
60 |
Cache TTL for RunPod cost lookups used by guardrail decisions. |
ISIS_RUNPOD_BUDGET_RETRY_AFTER_SECONDS |
60 |
Retry hint returned for hard-limit block responses. |
ISIS_RUNPOD_BUDGET_FAIL_CLOSED |
false |
If true, unavailable cost telemetry blocks requests instead of failing open. |
Rollout Procedure#
- Choose thresholds from forecast and risk envelope.
- Start in staging with conservative limits:
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=100
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=150
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=120
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=60
- Deploy generation API.
- Run guardrail verification script (below).
- Promote to production after staging verification is green.
Verification Procedure#
Run verification script:
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh
The script validates:
GET /api/v1/jobs/runpod/operationsis reachable.runpod.guardrail.enabled=trueis present in operations payload.- Synthetic submit either succeeds under guardrail headers or is correctly
throttled with
RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED.
Report output is written to tmp/isis-runpod-budget-guardrails/.
Alert and Triage Expectations#
- Guardrail mode should be monitored from
runpod.guardrail.modein operations payload and dashboards. - Sustained
throttle/blockmode requires incident triage for forecast correction, workload shaping, or provider routing adjustment.
Rollback#
Disable guardrail and redeploy generation API:
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=false
Use rollback only with explicit risk acceptance when spend controls are managed through another active mechanism.
Related Artifacts#
apps/isis/generation-api/src/middleware/runpod-budget-guardrail.tsapps/isis/generation-api/src/routes/jobs.tsscripts/isis/verify_runpod_budget_guardrails.shdocs/domains/isis/runbooks/runpod-oncall.md