# Isis SLO Definitions

Date: 2026-02-22  
Scope: Isis generation production path
(`submit -> queue -> execute -> complete`)  
Owner: `@oshun/platform` (with `@oshun/isis-team`)

## Purpose

Define production SLOs and SLI formulas for:

1. Submit latency
2. Queue delay
3. Execution success rate
4. End-to-end completion time

These definitions are the source-of-truth targets for dashboards, alerts, and
release-go/no-go checks.

## Measurement Policy

- Compliance window: rolling 30 days
- Evaluation interval: 1 minute (with 5m/1h burn-rate alerting windows)
- Exclusions:
  - Explicit user cancellations (`status=cancelled` with user cancel reason)
  - Planned maintenance windows marked by deployment/incident annotations
  - Synthetic test traffic labeled `metadata.synthetic=true`

## SLO Catalog

| SLO ID                        | Objective                                               | Target   |
| ----------------------------- | ------------------------------------------------------- | -------- |
| `isis.submit-latency`         | Good `POST /api/v1/jobs` requests complete in <= 2000ms | >= 99.0% |
| `isis.queue-delay`            | Jobs move from `queued` to `running` in <= 120s         | >= 97.0% |
| `isis.execution-success-rate` | Non-user-cancelled terminal jobs finish `completed`     | >= 97.0% |
| `isis.e2e-completion-time`    | Jobs complete in <= 900s from submit time               | >= 95.0% |

## SLI Definitions

### 1) Submit Latency (`isis.submit-latency`)

- Good event:
  - `POST /api/v1/jobs`
  - HTTP status `< 500`
  - request latency `<= 2000ms`
- Total event:
  - `POST /api/v1/jobs`
  - HTTP status `< 500`
- SLI:
  - `good_submit_requests / total_submit_requests`

### 2) Queue Delay (`isis.queue-delay`)

- Good event:
  - Job transitions to `running`
  - `running_at - queued_at <= 120s`
- Total event:
  - Jobs that entered `queued`
  - Excluding user-cancelled before start and synthetic traffic
- SLI:
  - `good_queue_delay_jobs / total_queued_jobs`

### 3) Execution Success Rate (`isis.execution-success-rate`)

- Good event:
  - Terminal status `completed`
- Total event:
  - Terminal statuses `completed | failed | timeout | provider_error`
  - Excluding explicit user cancellations
- SLI:
  - `completed_jobs / terminal_jobs_excluding_user_cancel`

### 4) End-to-End Completion (`isis.e2e-completion-time`)

- Good event:
  - Terminal status `completed`
  - `completed_at - submitted_at <= 900s`
- Total event:
  - Jobs accepted by `POST /api/v1/jobs`
  - Excluding explicit user cancellations and synthetic traffic
- SLI:
  - `good_e2e_jobs / total_accepted_jobs`

## Error Budgets (30 Days)

| SLO ID                        | Target | Error Budget |
| ----------------------------- | ------ | ------------ |
| `isis.submit-latency`         | 99.0%  | 1.0%         |
| `isis.queue-delay`            | 97.0%  | 3.0%         |
| `isis.execution-success-rate` | 97.0%  | 3.0%         |
| `isis.e2e-completion-time`    | 95.0%  | 5.0%         |

## Burn-Rate Alert Policy

For each SLO:

- Fast-burn page:
  - Burn rate >= 14 over 5 minutes
- Slow-burn ticket:
  - Burn rate >= 2 over 1 hour

Recommended severity mapping:

- `SEV-1`: Fast-burn on `execution-success-rate` or `e2e-completion-time`
- `SEV-2`: Fast-burn on `submit-latency` or `queue-delay`
- `SEV-3`: Slow-burn on any SLO

## Canonical Metric Contracts (for OBS-002 instrumentation)

- `isis_generation_submit_latency_ms` (histogram)
  - labels: `route`, `status_class`, `project_id`
- `isis_generation_queue_delay_seconds` (histogram)
  - labels: `priority`, `workflow_type`, `project_id`
- `isis_generation_terminal_total` (counter)
  - labels: `status`, `provider`, `workflow_type`, `project_id`
- `isis_generation_e2e_completion_seconds` (histogram)
  - labels: `provider`, `workflow_type`, `project_id`

## Release and Rollback Integration

- Release gate evidence (`QG-006`) must include current 7d/30d SLO compliance.
- Auto-rollback tie-in:
  - Submit-latency degradation threshold in
    `docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md`
  - Queue backlog and terminal failure regressions tied to SLO burn behavior.
- On-call playbook integration:
  - `docs/domains/isis/runbooks/creative-pipeline-oncall.md`
  - `scripts/isis/run_creative_pipeline_oncall_triage.sh`

## Traceability

- Checklist: `OBS-001`
- Next tasks:
  - `OBS-002` (metrics implementation)
  - `OBS-003` (provider latency/failure dashboards)
  - `OBS-005` (anomaly alerts)
