# Isis RunPod Budget Guardrails and Automatic Throttling Runbook

Last Updated: 2026-02-22  
Scope: Automatic cost guardrails for RunPod-backed generation traffic  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`, `@oshun/ml-team`

## Purpose

Define deterministic controls that automatically throttle generation submissions
when RunPod spend crosses configured thresholds.

This runbook covers:

1. Guardrail configuration for soft/hard spend thresholds.
2. Automatic throttling behavior for submit, batch submit, and retry flows.
3. Verification procedure for guardrail status and enforcement.

## Runtime Behavior Summary

When enabled, generation API evaluates RunPod monthly cost telemetry before
accepting expensive write-path actions (`POST /api/v1/jobs`,
`POST /api/v1/jobs/batch`, `POST /api/v1/jobs/:id/retry`):

- below soft limit: request allowed.
- at/above soft limit: request path enters throttled mode and uses a global
  throttle units window.
- at/above hard limit: request is rejected with `429` and
  `RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED`.

Guardrail status is exposed in `GET /api/v1/jobs/runpod/operations` at
`runpod.guardrail`.

## Configuration

| Environment Variable                           | Default | Description                                                                    |
| ---------------------------------------------- | ------- | ------------------------------------------------------------------------------ |
| `ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED`         | `false` | Enable automatic RunPod spend guardrails on write-path generation actions.     |
| `ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD`            | `0`     | Monthly estimated spend threshold that activates throttle mode.                |
| `ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD`            | `0`     | Monthly estimated spend threshold that blocks requests immediately.            |
| `ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW` | `120`   | Max weighted guardrail units allowed per throttle window in soft-limit mode.   |
| `ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS`   | `60`    | Duration of throttle window in seconds.                                        |
| `ISIS_RUNPOD_BUDGET_COST_CACHE_SECONDS`        | `60`    | Cache TTL for RunPod cost lookups used by guardrail decisions.                 |
| `ISIS_RUNPOD_BUDGET_RETRY_AFTER_SECONDS`       | `60`    | Retry hint returned for hard-limit block responses.                            |
| `ISIS_RUNPOD_BUDGET_FAIL_CLOSED`               | `false` | If `true`, unavailable cost telemetry blocks requests instead of failing open. |

## Rollout Procedure

1. Choose thresholds from forecast and risk envelope.
2. Start in staging with conservative limits:

```bash
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=100
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=150
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=120
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=60
```

3. Deploy generation API.
4. Run guardrail verification script (below).
5. Promote to production after staging verification is green.

## Verification Procedure

Run verification script:

```bash
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh
```

The script validates:

1. `GET /api/v1/jobs/runpod/operations` is reachable.
2. `runpod.guardrail.enabled=true` is present in operations payload.
3. Synthetic submit either succeeds under guardrail headers or is correctly
   throttled with `RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED`.

Report output is written to `tmp/isis-runpod-budget-guardrails/`.

## Alert and Triage Expectations

- Guardrail mode should be monitored from `runpod.guardrail.mode` in operations
  payload and dashboards.
- Sustained `throttle`/`block` mode requires incident triage for forecast
  correction, workload shaping, or provider routing adjustment.

## Rollback

Disable guardrail and redeploy generation API:

```bash
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=false
```

Use rollback only with explicit risk acceptance when spend controls are managed
through another active mechanism.

## Related Artifacts

- `apps/isis/generation-api/src/middleware/runpod-budget-guardrail.ts`
- `apps/isis/generation-api/src/routes/jobs.ts`
- `scripts/isis/verify_runpod_budget_guardrails.sh`
- `docs/domains/isis/runbooks/runpod-oncall.md`
