# ISIS v2 WS09-008: RunPod Endpoint Canary Strategy and Rollback Triggers

## Scope

Implemented automated endpoint-family canary strategy evaluation with rollback
triggers, exposed through RunPod operations API/UI telemetry and an execution
script for operational callback automation.

## Implemented Artifacts

- Canary strategy computation and API exposure:
  - `apps/isis/generation-api/src/routes/jobs.ts`
  - `apps/isis/generation-api/src/routes/jobs.spec.ts`
- Client + web telemetry surfaces:
  - `libs/isis/client/src/types.ts`
  - `libs/isis/client/src/resources/generation.ts`
  - `libs/isis/client/src/index.ts`
  - `apps/isis/web/src/pages/RunPodPage.tsx`
  - `apps/isis/web/src/pages/RunPodPage.spec.tsx`
- Operational execution script:
  - `scripts/isis/execute_runpod_endpoint_canary_strategy.sh`

## Strategy Model

- Strategy is computed per RunPod endpoint family using:
  - active canary percentage and endpoint IDs
  - canary endpoint SLO sample size and breach state
  - endpoint-level anomaly alerts
  - global cost-spike anomaly signal
- Actions emitted per family:
  - `promote` to the next ramp percentage
  - `hold` when gates are inconclusive (for example insufficient samples)
  - `rollback` when SLO/anomaly rollback triggers fire
  - `steady` when already at 100% and stable
- Rollback triggers are explicit (`rollbackTriggered=true`) and reason-coded.

## Verification

```bash
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
FAIL_ON_ROLLBACK=false OPERATIONS_FILE=/tmp/runpod-operations.json scripts/isis/execute_runpod_endpoint_canary_strategy.sh
```

## Outcome

WS09-008 is now closed with deterministic, automated canary actioning and
rollback trigger signaling for endpoint-family upgrades, including operational
callback execution support.
