Last Updated: 2026-02-22
Scope: RunPod API key and webhook secret used by Isis generation services
Primary Owners: @oshun/devops, @oshun/isis-team
Purpose#
Provide deterministic rotation procedures for:
RUNPOD_API_KEYcredential used for RunPod endpoint operations.RUNPOD_WEBHOOK_SECRETused to validatePOST /webhooks/runpodcallback signatures.
This runbook applies to all environments with RunPod enabled and references
Terraform secret resources in infra/terraform/secrets/runpod.tf.
Secret Inventory#
Default AWS Secrets Manager paths:
/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api(api_key)/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook(webhook_secret)
Rotation Cadence and Triggers#
| Secret | Standard Cadence | Emergency Triggers |
|---|---|---|
RUNPOD_API_KEY |
Every 30 days | suspected credential leak, unauthorized usage, vendor security notice |
RUNPOD_WEBHOOK_SECRET |
Every 30 days | signature mismatch incident, callback tampering suspicion, compromise investigation |
Rotate immediately for any confirmed compromise.
Preconditions#
- AWS IAM access to Secrets Manager and ECS update-service.
- RunPod console access for API key and webhook endpoint settings.
- Verified service ownership for:
- generation API deployment
- RunPod webhook endpoint configuration.
- Incident/change ticket opened.
Rotation Procedure#
A. Pre-Rotation Snapshot#
- Capture current versions (no secret values):
bash
aws secretsmanager list-secret-version-ids \
--secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
--query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'
aws secretsmanager list-secret-version-ids \
--secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
--query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'
- Record current Isis service health:
bash
curl -s "${ISIS_API_BASE_URL}/health" | jq
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq
B. Rotate Secrets (Automation)#
Dry-run first:
bash
DRY_RUN=true \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh
Execute rotation:
bash
DRY_RUN=false \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh
Automation outputs a report in tmp/isis-secrets-rotation/.
C. Vendor-Side Sync#
- Ensure RunPod console has the newly generated API key active.
- Update RunPod webhook configuration to use the new webhook secret.
- Send a signed callback test (or provider test event) after both sides update.
D. Post-Rotation Verification#
- Verify generation API has healthy RunPod telemetry:
bash
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq '.runpod'
- Verify webhook signature acceptance:
bash
PAYLOAD='{"id":"rotation_test","status":"COMPLETED"}'
SIG="$(printf '%s' "${PAYLOAD}" | openssl dgst -sha256 -hmac "${RUNPOD_WEBHOOK_SECRET_NEW}" -binary | xxd -p -c 256)"
curl -sS -X POST "${ISIS_API_BASE_URL}/webhooks/runpod" \
-H "Content-Type: application/json" \
-H "X-RunPod-Signature: ${SIG}" \
-d "${PAYLOAD}" | jq
Expected status: 202 with accepted: true.
- Submit one synthetic generation and confirm completion path.
Rollback Procedure#
If post-rotation verification fails:
- Re-promote previous secret versions:
bash
aws secretsmanager update-secret-version-stage \
--secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
--version-stage AWSCURRENT \
--move-to-version-id "<previous-version-id>"
aws secretsmanager update-secret-version-stage \
--secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
--version-stage AWSCURRENT \
--move-to-version-id "<previous-version-id>"
- Revert RunPod console webhook secret/API key to previous values.
- Force new deployment of generation API.
- Re-run post-rotation verification checks.
Evidence Checklist#
- Rotation report file from
scripts/isis/rotate_runpod_credentials.sh. - Secret version IDs before and after rotation.
- Post-rotation RunPod operations snapshot.
- Signed webhook callback verification result.
- Incident/change ticket with completion timestamp.
Related Artifacts#
scripts/isis/rotate_runpod_credentials.shinfra/terraform/secrets/runpod.tfapps/isis/generation-api/src/routes/provider-callbacks.tsdocs/domains/isis/runbooks/runpod-oncall.md