# Isis RunPod Secret Rotation Runbook

Last Updated: 2026-02-22  
Scope: RunPod API key and webhook secret used by Isis generation services  
Primary Owners: `@oshun/devops`, `@oshun/isis-team`

## Purpose

Provide deterministic rotation procedures for:

1. `RUNPOD_API_KEY` credential used for RunPod endpoint operations.
2. `RUNPOD_WEBHOOK_SECRET` used to validate `POST /webhooks/runpod` callback
   signatures.

This runbook applies to all environments with RunPod enabled and references
Terraform secret resources in `infra/terraform/secrets/runpod.tf`.

## Secret Inventory

Default AWS Secrets Manager paths:

- `/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api` (`api_key`)
- `/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook` (`webhook_secret`)

## Rotation Cadence and Triggers

| Secret                  | Standard Cadence | Emergency Triggers                                                                  |
| ----------------------- | ---------------- | ----------------------------------------------------------------------------------- |
| `RUNPOD_API_KEY`        | Every 30 days    | suspected credential leak, unauthorized usage, vendor security notice               |
| `RUNPOD_WEBHOOK_SECRET` | Every 30 days    | signature mismatch incident, callback tampering suspicion, compromise investigation |

Rotate immediately for any confirmed compromise.

## Preconditions

- AWS IAM access to Secrets Manager and ECS update-service.
- RunPod console access for API key and webhook endpoint settings.
- Verified service ownership for:
  - generation API deployment
  - RunPod webhook endpoint configuration.
- Incident/change ticket opened.

## Rotation Procedure

### A. Pre-Rotation Snapshot

1. Capture current versions (no secret values):

```bash
aws secretsmanager list-secret-version-ids \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
  --query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'

aws secretsmanager list-secret-version-ids \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
  --query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'
```

2. Record current Isis service health:

```bash
curl -s "${ISIS_API_BASE_URL}/health" | jq
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq
```

### B. Rotate Secrets (Automation)

Dry-run first:

```bash
DRY_RUN=true \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh
```

Execute rotation:

```bash
DRY_RUN=false \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh
```

Automation outputs a report in `tmp/isis-secrets-rotation/`.

### C. Vendor-Side Sync

1. Ensure RunPod console has the newly generated API key active.
2. Update RunPod webhook configuration to use the new webhook secret.
3. Send a signed callback test (or provider test event) after both sides update.

### D. Post-Rotation Verification

1. Verify generation API has healthy RunPod telemetry:

```bash
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq '.runpod'
```

2. Verify webhook signature acceptance:

```bash
PAYLOAD='{"id":"rotation_test","status":"COMPLETED"}'
SIG="$(printf '%s' "${PAYLOAD}" | openssl dgst -sha256 -hmac "${RUNPOD_WEBHOOK_SECRET_NEW}" -binary | xxd -p -c 256)"

curl -sS -X POST "${ISIS_API_BASE_URL}/webhooks/runpod" \
  -H "Content-Type: application/json" \
  -H "X-RunPod-Signature: ${SIG}" \
  -d "${PAYLOAD}" | jq
```

Expected status: `202` with `accepted: true`.

3. Submit one synthetic generation and confirm completion path.

## Rollback Procedure

If post-rotation verification fails:

1. Re-promote previous secret versions:

```bash
aws secretsmanager update-secret-version-stage \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
  --version-stage AWSCURRENT \
  --move-to-version-id "<previous-version-id>"

aws secretsmanager update-secret-version-stage \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
  --version-stage AWSCURRENT \
  --move-to-version-id "<previous-version-id>"
```

2. Revert RunPod console webhook secret/API key to previous values.
3. Force new deployment of generation API.
4. Re-run post-rotation verification checks.

## Evidence Checklist

- Rotation report file from `scripts/isis/rotate_runpod_credentials.sh`.
- Secret version IDs before and after rotation.
- Post-rotation RunPod operations snapshot.
- Signed webhook callback verification result.
- Incident/change ticket with completion timestamp.

## Related Artifacts

- `scripts/isis/rotate_runpod_credentials.sh`
- `infra/terraform/secrets/runpod.tf`
- `apps/isis/generation-api/src/routes/provider-callbacks.ts`
- `docs/domains/isis/runbooks/runpod-oncall.md`
