Eve library

ISIS Canary SLO and Cost Verification (REL-003)

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: REL-003
Ticket: ISIS-0177

Purpose#

Provide a deterministic verification gate for canary rollout quality by checking canary traffic against SLO and cost guardrails before ramping traffic.

This verification uses historical job records (jobs table) and canary routing metadata (metadata.queueDispatch) produced by queue rollout controls.

Verification Script#

  • Script: scripts/isis/verify_canary_slo_cost.sh
  • Requirements: psql, jq, and ISIS_DATABASE_URL (or DATABASE_URL)
  • Output: JSON report in tmp/isis-canary/ by default
  • Ramp controller integration: scripts/isis/advance_queue_canary_rollout.sh

Default Usage#

bash
ISIS_DATABASE_URL=postgres://... \
  scripts/isis/verify_canary_slo_cost.sh

Strict Gate Usage (Fail Process on Breach)#

bash
ISIS_DATABASE_URL=postgres://... \
FAIL_ON_BREACH=true \
LOOKBACK_HOURS=24 \
BASELINE_HOURS=24 \
MIN_CANARY_TERMINAL_JOBS=30 \
scripts/isis/verify_canary_slo_cost.sh

Include Submit-Latency SLO from External Metrics#

Submit latency is not derivable from persisted jobs records. Provide the SLI from dashboards/metrics when required:

bash
ISIS_DATABASE_URL=postgres://... \
REQUIRE_SUBMIT_SLI=true \
SUBMIT_LATENCY_SLI=0.992 \
scripts/isis/verify_canary_slo_cost.sh

Cohort and Window Semantics#

  • Canary cohort:
    • metadata.queueDispatch.mode=canary
    • metadata.queueDispatch.rolloutEligible=true
  • Control cohort:
    • all non-canary-eligible traffic in the same lookback window
  • Synthetic traffic:
    • excluded when metadata.synthetic=true
  • Current window:
    • LOOKBACK_HOURS (default 24)
  • Baseline window:
    • preceding BASELINE_HOURS window (default 24)

Metrics Calculated#

Per cohort (canary, control) and baseline:

  • Volume: totalJobs, terminalJobs, completedJobs, failedJobs
  • Queue SLO signals:
    • queueDelaySli (share with started_at - created_at <= 120s)
    • queueDelayP95Seconds
  • Execution SLO signals:
    • executionSuccessRate (completed / terminal)
    • e2eCompletionSli (share with completed_at - created_at <= 900s)
  • Cost quality signals:
    • costCompletenessRate (completed jobs with both cost and runtime fields)
    • averageCostUsd, costP95Usd, totalCostUsd
    • costPerRuntimeSecondUsd

Cost and runtime aliases are resolved from existing Isis metadata conventions:

  • cost: output.metadata.estimatedCostUsd, output.metadata.costUsd, metadata.estimatedCostUsd, metadata.costUsd, metadata.providerExecution.estimatedCostUsd, metadata.costBackfill.estimatedCostUsd
  • runtime: output.metadata.executionTimeMs, metadata.providerExecution.executionTimeMs, metadata.costBackfill.executionTimeMs, and runtimeSeconds aliases

Gate Checks and Defaults#

The script emits .checks and .overall.pass using these defaults:

  • sampleSize: canary terminal jobs >= 30
  • queueDelaySli: >= 0.97
  • executionSuccessRate: >= 0.97
  • e2eCompletionSli: >= 0.95
  • costCompletenessRate: >= 0.995
  • canaryFailureRate: <= 0.10
  • queueDelayP95Regression: <= 1.0 (no more than +100% regression)
  • averageCostRegression: <= 0.20 (no more than +20% regression)
  • submitLatencySli: optional unless REQUIRE_SUBMIT_SLI=true

These align with rollback criteria and release quality bar requirements:

  • docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
  • docs/domains/isis/remediation/ISIS_RELEASE_QUALITY_BAR.md
  • docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md

Output Contract (High Level)#

json
{
  "generatedAt": "...",
  "lookbackHours": 24,
  "baselineHours": 24,
  "cohorts": {
    "canary": { "...": "..." },
    "control": { "...": "..." }
  },
  "baseline": { "...": "..." },
  "thresholds": { "...": "..." },
  "comparison": { "...": "..." },
  "checks": {
    "sampleSize": { "pass": true },
    "queueDelaySli": { "pass": true },
    "...": { "pass": false }
  },
  "overall": {
    "pass": false,
    "failedChecks": ["averageCostRegression"]
  }
}

Release Decision Guidance#

  • overall.pass=true:
    • canary SLO/cost signals are within configured thresholds
    • eligible to proceed to planned traffic increase (subject to full sign-offs)
  • overall.pass=false:
    • hold rollout and investigate failed checks
    • if criteria map to rollback triggers, reduce canary to 0% and execute rollback procedures

Traceability#

  • REL-001: queue producer global rollout flag
  • REL-002: project/percentage canary routing
  • REL-003: SLO/cost verification during canary (this artifact)
  • REL-004: automated ramp/rollback decisioning consumes this report