Eve library

ISIS v2 WS09-008: RunPod Endpoint Canary Strategy and Rollback Triggers

Guide

Revision 1

On this page

Scope#

Implemented automated endpoint-family canary strategy evaluation with rollback triggers, exposed through RunPod operations API/UI telemetry and an execution script for operational callback automation.

Implemented Artifacts#

  • Canary strategy computation and API exposure:
    • apps/isis/generation-api/src/routes/jobs.ts
    • apps/isis/generation-api/src/routes/jobs.spec.ts
  • Client + web telemetry surfaces:
    • libs/isis/client/src/types.ts
    • libs/isis/client/src/resources/generation.ts
    • libs/isis/client/src/index.ts
    • apps/isis/web/src/pages/RunPodPage.tsx
    • apps/isis/web/src/pages/RunPodPage.spec.tsx
  • Operational execution script:
    • scripts/isis/execute_runpod_endpoint_canary_strategy.sh

Strategy Model#

  • Strategy is computed per RunPod endpoint family using:
    • active canary percentage and endpoint IDs
    • canary endpoint SLO sample size and breach state
    • endpoint-level anomaly alerts
    • global cost-spike anomaly signal
  • Actions emitted per family:
    • promote to the next ramp percentage
    • hold when gates are inconclusive (for example insufficient samples)
    • rollback when SLO/anomaly rollback triggers fire
    • steady when already at 100% and stable
  • Rollback triggers are explicit (rollbackTriggered=true) and reason-coded.

Verification#

bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
FAIL_ON_ROLLBACK=false OPERATIONS_FILE=/tmp/runpod-operations.json scripts/isis/execute_runpod_endpoint_canary_strategy.sh

Outcome#

WS09-008 is now closed with deterministic, automated canary actioning and rollback trigger signaling for endpoint-family upgrades, including operational callback execution support.