Eve library

Isis RunPod Secret Rotation Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-22
Scope: RunPod API key and webhook secret used by Isis generation services
Primary Owners: @oshun/devops, @oshun/isis-team

Purpose#

Provide deterministic rotation procedures for:

  1. RUNPOD_API_KEY credential used for RunPod endpoint operations.
  2. RUNPOD_WEBHOOK_SECRET used to validate POST /webhooks/runpod callback signatures.

This runbook applies to all environments with RunPod enabled and references Terraform secret resources in infra/terraform/secrets/runpod.tf.

Secret Inventory#

Default AWS Secrets Manager paths:

  • /${NAME_PREFIX}/${ENVIRONMENT}/runpod/api (api_key)
  • /${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook (webhook_secret)

Rotation Cadence and Triggers#

Secret Standard Cadence Emergency Triggers
RUNPOD_API_KEY Every 30 days suspected credential leak, unauthorized usage, vendor security notice
RUNPOD_WEBHOOK_SECRET Every 30 days signature mismatch incident, callback tampering suspicion, compromise investigation

Rotate immediately for any confirmed compromise.

Preconditions#

  • AWS IAM access to Secrets Manager and ECS update-service.
  • RunPod console access for API key and webhook endpoint settings.
  • Verified service ownership for:
    • generation API deployment
    • RunPod webhook endpoint configuration.
  • Incident/change ticket opened.

Rotation Procedure#

A. Pre-Rotation Snapshot#

  1. Capture current versions (no secret values):
bash
aws secretsmanager list-secret-version-ids \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
  --query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'

aws secretsmanager list-secret-version-ids \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
  --query '{Versions:Versions[].{VersionId:VersionId,Stages:VersionStages}}'
  1. Record current Isis service health:
bash
curl -s "${ISIS_API_BASE_URL}/health" | jq
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq

B. Rotate Secrets (Automation)#

Dry-run first:

bash
DRY_RUN=true \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh

Execute rotation:

bash
DRY_RUN=false \
NAME_PREFIX=oshun \
ENVIRONMENT=production \
RUNPOD_API_KEY_NEW="rp_live_new_..." \
RUNPOD_WEBHOOK_SECRET_NEW="whsec_new_..." \
FORCE_DEPLOY=true \
ECS_CLUSTER_NAME="oshun-production" \
ISIS_GENERATION_SERVICE_NAME="oshun-production-isis-generation-api" \
scripts/isis/rotate_runpod_credentials.sh

Automation outputs a report in tmp/isis-secrets-rotation/.

C. Vendor-Side Sync#

  1. Ensure RunPod console has the newly generated API key active.
  2. Update RunPod webhook configuration to use the new webhook secret.
  3. Send a signed callback test (or provider test event) after both sides update.

D. Post-Rotation Verification#

  1. Verify generation API has healthy RunPod telemetry:
bash
curl -s "${ISIS_API_BASE_URL}/api/v1/jobs/runpod/operations" | jq '.runpod'
  1. Verify webhook signature acceptance:
bash
PAYLOAD='{"id":"rotation_test","status":"COMPLETED"}'
SIG="$(printf '%s' "${PAYLOAD}" | openssl dgst -sha256 -hmac "${RUNPOD_WEBHOOK_SECRET_NEW}" -binary | xxd -p -c 256)"

curl -sS -X POST "${ISIS_API_BASE_URL}/webhooks/runpod" \
  -H "Content-Type: application/json" \
  -H "X-RunPod-Signature: ${SIG}" \
  -d "${PAYLOAD}" | jq

Expected status: 202 with accepted: true.

  1. Submit one synthetic generation and confirm completion path.

Rollback Procedure#

If post-rotation verification fails:

  1. Re-promote previous secret versions:
bash
aws secretsmanager update-secret-version-stage \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/api" \
  --version-stage AWSCURRENT \
  --move-to-version-id "<previous-version-id>"

aws secretsmanager update-secret-version-stage \
  --secret-id "/${NAME_PREFIX}/${ENVIRONMENT}/runpod/webhook" \
  --version-stage AWSCURRENT \
  --move-to-version-id "<previous-version-id>"
  1. Revert RunPod console webhook secret/API key to previous values.
  2. Force new deployment of generation API.
  3. Re-run post-rotation verification checks.

Evidence Checklist#

  • Rotation report file from scripts/isis/rotate_runpod_credentials.sh.
  • Secret version IDs before and after rotation.
  • Post-rotation RunPod operations snapshot.
  • Signed webhook callback verification result.
  • Incident/change ticket with completion timestamp.
  • scripts/isis/rotate_runpod_credentials.sh
  • infra/terraform/secrets/runpod.tf
  • apps/isis/generation-api/src/routes/provider-callbacks.ts
  • docs/domains/isis/runbooks/runpod-oncall.md