Scope#
Implemented automated endpoint-family canary strategy evaluation with rollback triggers, exposed through RunPod operations API/UI telemetry and an execution script for operational callback automation.
Implemented Artifacts#
- Canary strategy computation and API exposure:
apps/isis/generation-api/src/routes/jobs.tsapps/isis/generation-api/src/routes/jobs.spec.ts
- Client + web telemetry surfaces:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.tslibs/isis/client/src/index.tsapps/isis/web/src/pages/RunPodPage.tsxapps/isis/web/src/pages/RunPodPage.spec.tsx
- Operational execution script:
scripts/isis/execute_runpod_endpoint_canary_strategy.sh
Strategy Model#
- Strategy is computed per RunPod endpoint family using:
- active canary percentage and endpoint IDs
- canary endpoint SLO sample size and breach state
- endpoint-level anomaly alerts
- global cost-spike anomaly signal
- Actions emitted per family:
promoteto the next ramp percentageholdwhen gates are inconclusive (for example insufficient samples)rollbackwhen SLO/anomaly rollback triggers firesteadywhen already at 100% and stable
- Rollback triggers are explicit (
rollbackTriggered=true) and reason-coded.
Verification#
bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
FAIL_ON_ROLLBACK=false OPERATIONS_FILE=/tmp/runpod-operations.json scripts/isis/execute_runpod_endpoint_canary_strategy.sh
Outcome#
WS09-008 is now closed with deterministic, automated canary actioning and rollback trigger signaling for endpoint-family upgrades, including operational callback execution support.