# OPS-004 Certification: RunPod Budget Guardrails and Automatic Throttling

Date: 2026-02-22  
Checklist Item: `OPS-004`  
Ticket: `ISIS-0239`

## Scope

Operational cost-protection coverage for RunPod-backed generation paths:

1. Threshold-based budget guardrails (soft/hard spend).
2. Automatic throttling on generation write-path actions.
3. Operator visibility of guardrail state in RunPod operations payload.
4. Deterministic verification procedure and runbook controls.

## Evidence Matrix

| Requirement                                                  | Evidence                                                                                                                                                                                                                                 | Status |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
| Budget guardrail logic exists in production write path       | `apps/isis/generation-api/src/middleware/runpod-budget-guardrail.ts` implements threshold evaluation, cached cost snapshot ingestion, soft-limit throttle windows, hard-limit blocking, and deterministic headers.                       | Pass   |
| Submit/batch/retry endpoints enforce guardrail automatically | `apps/isis/generation-api/src/routes/jobs.ts` evaluates guardrail before abuse-control checks for `POST /jobs`, `POST /jobs/batch`, and `POST /jobs/:id/retry`, returning `429` + `RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED` when blocked. | Pass   |
| Guardrail status is operationally visible                    | Same route file exposes guardrail state in `GET /api/v1/jobs/runpod/operations` as `runpod.guardrail`, including mode, thresholds, throttle window usage, and cost-source metadata.                                                      | Pass   |
| Runtime enforcement has automated tests                      | `apps/isis/generation-api/src/routes/jobs.spec.ts` includes hard-limit block and soft-limit throttle exhaustion tests; `apps/isis/generation-api/src/middleware/runpod-budget-guardrail.spec.ts` covers core decision logic.             | Pass   |
| Runbook exists with deployment and rollback controls         | `docs/domains/isis/runbooks/runpod-budget-guardrails.md` defines env controls, rollout sequence, verification commands, alert expectations, and rollback procedure.                                                                      | Pass   |
| Verification automation exists                               | `scripts/isis/verify_runpod_budget_guardrails.sh` validates operations payload guardrail state and synthetic submit behavior (allowed vs correctly throttled) with machine-readable JSON report output.                                  | Pass   |

## Reproduction Commands

```bash
pnpm --filter @isis/generation-api exec vitest run src/middleware/runpod-budget-guardrail.spec.ts src/routes/jobs.spec.ts
bash -n scripts/isis/verify_runpod_budget_guardrails.sh

STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh
```
