Date: 2026-02-22
Checklist Item: OPS-004
Ticket: ISIS-0239
Scope#
Operational cost-protection coverage for RunPod-backed generation paths:
- Threshold-based budget guardrails (soft/hard spend).
- Automatic throttling on generation write-path actions.
- Operator visibility of guardrail state in RunPod operations payload.
- Deterministic verification procedure and runbook controls.
Evidence Matrix#
| Requirement | Evidence | Status |
|---|---|---|
| Budget guardrail logic exists in production write path | apps/isis/generation-api/src/middleware/runpod-budget-guardrail.ts implements threshold evaluation, cached cost snapshot ingestion, soft-limit throttle windows, hard-limit blocking, and deterministic headers. |
Pass |
| Submit/batch/retry endpoints enforce guardrail automatically | apps/isis/generation-api/src/routes/jobs.ts evaluates guardrail before abuse-control checks for POST /jobs, POST /jobs/batch, and POST /jobs/:id/retry, returning 429 + RUNPOD_BUDGET_GUARDRAIL_LIMIT_EXCEEDED when blocked. |
Pass |
| Guardrail status is operationally visible | Same route file exposes guardrail state in GET /api/v1/jobs/runpod/operations as runpod.guardrail, including mode, thresholds, throttle window usage, and cost-source metadata. |
Pass |
| Runtime enforcement has automated tests | apps/isis/generation-api/src/routes/jobs.spec.ts includes hard-limit block and soft-limit throttle exhaustion tests; apps/isis/generation-api/src/middleware/runpod-budget-guardrail.spec.ts covers core decision logic. |
Pass |
| Runbook exists with deployment and rollback controls | docs/domains/isis/runbooks/runpod-budget-guardrails.md defines env controls, rollout sequence, verification commands, alert expectations, and rollback procedure. |
Pass |
| Verification automation exists | scripts/isis/verify_runpod_budget_guardrails.sh validates operations payload guardrail state and synthetic submit behavior (allowed vs correctly throttled) with machine-readable JSON report output. |
Pass |
Reproduction Commands#
bash
pnpm --filter @isis/generation-api exec vitest run src/middleware/runpod-budget-guardrail.spec.ts src/routes/jobs.spec.ts
bash -n scripts/isis/verify_runpod_budget_guardrails.sh
STRICT=true \
ISIS_API_BASE_URL="https://isis-staging.example.com" \
ISIS_ADMIN_TOKEN="<token>" \
scripts/isis/verify_runpod_budget_guardrails.sh