# ISIS Canary SLO and Cost Verification (REL-003)

Date: 2026-02-22  
Checklist Item: `REL-003`  
Ticket: `ISIS-0177`

## Purpose

Provide a deterministic verification gate for canary rollout quality by checking
canary traffic against SLO and cost guardrails before ramping traffic.

This verification uses historical job records (`jobs` table) and canary routing
metadata (`metadata.queueDispatch`) produced by queue rollout controls.

## Verification Script

- Script: `scripts/isis/verify_canary_slo_cost.sh`
- Requirements: `psql`, `jq`, and `ISIS_DATABASE_URL` (or `DATABASE_URL`)
- Output: JSON report in `tmp/isis-canary/` by default
- Ramp controller integration: `scripts/isis/advance_queue_canary_rollout.sh`

### Default Usage

```bash
ISIS_DATABASE_URL=postgres://... \
  scripts/isis/verify_canary_slo_cost.sh
```

### Strict Gate Usage (Fail Process on Breach)

```bash
ISIS_DATABASE_URL=postgres://... \
FAIL_ON_BREACH=true \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
MIN_CANARY_TERMINAL_JOBS=30 \
scripts/isis/verify_canary_slo_cost.sh
```

### Include Submit-Latency SLO from External Metrics

Submit latency is not derivable from persisted `jobs` records. Provide the SLI
from dashboards/metrics when required:

```bash
ISIS_DATABASE_URL=postgres://... \
REQUIRE_SUBMIT_SLI=true \
SUBMIT_LATENCY_SLI=0.992 \
scripts/isis/verify_canary_slo_cost.sh
```

## Cohort and Window Semantics

- Canary cohort:
  - `metadata.queueDispatch.mode=canary`
  - `metadata.queueDispatch.rolloutEligible=true`
- Control cohort:
  - all non-canary-eligible traffic in the same lookback window
- Synthetic traffic:
  - excluded when `metadata.synthetic=true`
- Current window:
  - `LOOKBACK_HOURS` (default `24`)
- Baseline window:
  - preceding `BASELINE_HOURS` window (default `24`)

## Metrics Calculated

Per cohort (`canary`, `control`) and baseline:

- Volume: `totalJobs`, `terminalJobs`, `completedJobs`, `failedJobs`
- Queue SLO signals:
  - `queueDelaySli` (share with `started_at - created_at <= 120s`)
  - `queueDelayP95Seconds`
- Execution SLO signals:
  - `executionSuccessRate` (`completed / terminal`)
  - `e2eCompletionSli` (share with `completed_at - created_at <= 900s`)
- Cost quality signals:
  - `costCompletenessRate` (completed jobs with both cost and runtime fields)
  - `averageCostUsd`, `costP95Usd`, `totalCostUsd`
  - `costPerRuntimeSecondUsd`

Cost and runtime aliases are resolved from existing Isis metadata conventions:

- cost: `output.metadata.estimatedCostUsd`, `output.metadata.costUsd`,
  `metadata.estimatedCostUsd`, `metadata.costUsd`,
  `metadata.providerExecution.estimatedCostUsd`,
  `metadata.costBackfill.estimatedCostUsd`
- runtime: `output.metadata.executionTimeMs`,
  `metadata.providerExecution.executionTimeMs`,
  `metadata.costBackfill.executionTimeMs`, and `runtimeSeconds` aliases

## Gate Checks and Defaults

The script emits `.checks` and `.overall.pass` using these defaults:

- `sampleSize`: canary terminal jobs >= `30`
- `queueDelaySli`: >= `0.97`
- `executionSuccessRate`: >= `0.97`
- `e2eCompletionSli`: >= `0.95`
- `costCompletenessRate`: >= `0.995`
- `canaryFailureRate`: <= `0.10`
- `queueDelayP95Regression`: <= `1.0` (no more than +100% regression)
- `averageCostRegression`: <= `0.20` (no more than +20% regression)
- `submitLatencySli`: optional unless `REQUIRE_SUBMIT_SLI=true`

These align with rollback criteria and release quality bar requirements:

- `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
- `docs/domains/isis/remediation/ISIS_RELEASE_QUALITY_BAR.md`
- `docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md`

## Output Contract (High Level)

```json
{
  "generatedAt": "...",
  "lookbackHours": 24,
  "baselineHours": 24,
  "cohorts": {
    "canary": { "...": "..." },
    "control": { "...": "..." }
  },
  "baseline": { "...": "..." },
  "thresholds": { "...": "..." },
  "comparison": { "...": "..." },
  "checks": {
    "sampleSize": { "pass": true },
    "queueDelaySli": { "pass": true },
    "...": { "pass": false }
  },
  "overall": {
    "pass": false,
    "failedChecks": ["averageCostRegression"]
  }
}
```

## Release Decision Guidance

- `overall.pass=true`:
  - canary SLO/cost signals are within configured thresholds
  - eligible to proceed to planned traffic increase (subject to full sign-offs)
- `overall.pass=false`:
  - hold rollout and investigate failed checks
  - if criteria map to rollback triggers, reduce canary to `0%` and execute
    rollback procedures

## Traceability

- `REL-001`: queue producer global rollout flag
- `REL-002`: project/percentage canary routing
- `REL-003`: SLO/cost verification during canary (this artifact)
- `REL-004`: automated ramp/rollback decisioning consumes this report
