Eve library

Isis SLO Definitions

Guide

Revision 1

On this page

Date: 2026-02-22
Scope: Isis generation production path (submit -> queue -> execute -> complete)
Owner: @oshun/platform (with @oshun/isis-team)

Purpose#

Define production SLOs and SLI formulas for:

  1. Submit latency
  2. Queue delay
  3. Execution success rate
  4. End-to-end completion time

These definitions are the source-of-truth targets for dashboards, alerts, and release-go/no-go checks.

Measurement Policy#

  • Compliance window: rolling 30 days
  • Evaluation interval: 1 minute (with 5m/1h burn-rate alerting windows)
  • Exclusions:
    • Explicit user cancellations (status=cancelled with user cancel reason)
    • Planned maintenance windows marked by deployment/incident annotations
    • Synthetic test traffic labeled metadata.synthetic=true

SLO Catalog#

SLO ID Objective Target
isis.submit-latency Good POST /api/v1/jobs requests complete in <= 2000ms >= 99.0%
isis.queue-delay Jobs move from queued to running in <= 120s >= 97.0%
isis.execution-success-rate Non-user-cancelled terminal jobs finish completed >= 97.0%
isis.e2e-completion-time Jobs complete in <= 900s from submit time >= 95.0%

SLI Definitions#

1) Submit Latency (isis.submit-latency)#

  • Good event:
    • POST /api/v1/jobs
    • HTTP status < 500
    • request latency <= 2000ms
  • Total event:
    • POST /api/v1/jobs
    • HTTP status < 500
  • SLI:
    • good_submit_requests / total_submit_requests

2) Queue Delay (isis.queue-delay)#

  • Good event:
    • Job transitions to running
    • running_at - queued_at <= 120s
  • Total event:
    • Jobs that entered queued
    • Excluding user-cancelled before start and synthetic traffic
  • SLI:
    • good_queue_delay_jobs / total_queued_jobs

3) Execution Success Rate (isis.execution-success-rate)#

  • Good event:
    • Terminal status completed
  • Total event:
    • Terminal statuses completed | failed | timeout | provider_error
    • Excluding explicit user cancellations
  • SLI:
    • completed_jobs / terminal_jobs_excluding_user_cancel

4) End-to-End Completion (isis.e2e-completion-time)#

  • Good event:
    • Terminal status completed
    • completed_at - submitted_at <= 900s
  • Total event:
    • Jobs accepted by POST /api/v1/jobs
    • Excluding explicit user cancellations and synthetic traffic
  • SLI:
    • good_e2e_jobs / total_accepted_jobs

Error Budgets (30 Days)#

SLO ID Target Error Budget
isis.submit-latency 99.0% 1.0%
isis.queue-delay 97.0% 3.0%
isis.execution-success-rate 97.0% 3.0%
isis.e2e-completion-time 95.0% 5.0%

Burn-Rate Alert Policy#

For each SLO:

  • Fast-burn page:
    • Burn rate >= 14 over 5 minutes
  • Slow-burn ticket:
    • Burn rate >= 2 over 1 hour

Recommended severity mapping:

  • SEV-1: Fast-burn on execution-success-rate or e2e-completion-time
  • SEV-2: Fast-burn on submit-latency or queue-delay
  • SEV-3: Slow-burn on any SLO

Canonical Metric Contracts (for OBS-002 instrumentation)#

  • isis_generation_submit_latency_ms (histogram)
    • labels: route, status_class, project_id
  • isis_generation_queue_delay_seconds (histogram)
    • labels: priority, workflow_type, project_id
  • isis_generation_terminal_total (counter)
    • labels: status, provider, workflow_type, project_id
  • isis_generation_e2e_completion_seconds (histogram)
    • labels: provider, workflow_type, project_id

Release and Rollback Integration#

  • Release gate evidence (QG-006) must include current 7d/30d SLO compliance.
  • Auto-rollback tie-in:
    • Submit-latency degradation threshold in docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md
    • Queue backlog and terminal failure regressions tied to SLO burn behavior.
  • On-call playbook integration:
    • docs/domains/isis/runbooks/creative-pipeline-oncall.md
    • scripts/isis/run_creative_pipeline_oncall_triage.sh

Traceability#

  • Checklist: OBS-001
  • Next tasks:
    • OBS-002 (metrics implementation)
    • OBS-003 (provider latency/failure dashboards)
    • OBS-005 (anomaly alerts)