Date: 2026-02-22
Scope: Isis generation production path
(submit -> queue -> execute -> complete)
Owner: @oshun/platform (with @oshun/isis-team)
Purpose#
Define production SLOs and SLI formulas for:
- Submit latency
- Queue delay
- Execution success rate
- End-to-end completion time
These definitions are the source-of-truth targets for dashboards, alerts, and release-go/no-go checks.
Measurement Policy#
- Compliance window: rolling 30 days
- Evaluation interval: 1 minute (with 5m/1h burn-rate alerting windows)
- Exclusions:
- Explicit user cancellations (
status=cancelledwith user cancel reason) - Planned maintenance windows marked by deployment/incident annotations
- Synthetic test traffic labeled
metadata.synthetic=true
- Explicit user cancellations (
SLO Catalog#
| SLO ID | Objective | Target |
|---|---|---|
isis.submit-latency |
Good POST /api/v1/jobs requests complete in <= 2000ms |
>= 99.0% |
isis.queue-delay |
Jobs move from queued to running in <= 120s |
>= 97.0% |
isis.execution-success-rate |
Non-user-cancelled terminal jobs finish completed |
>= 97.0% |
isis.e2e-completion-time |
Jobs complete in <= 900s from submit time | >= 95.0% |
SLI Definitions#
1) Submit Latency (isis.submit-latency)#
- Good event:
POST /api/v1/jobs- HTTP status
< 500 - request latency
<= 2000ms
- Total event:
POST /api/v1/jobs- HTTP status
< 500
- SLI:
good_submit_requests / total_submit_requests
2) Queue Delay (isis.queue-delay)#
- Good event:
- Job transitions to
running running_at - queued_at <= 120s
- Job transitions to
- Total event:
- Jobs that entered
queued - Excluding user-cancelled before start and synthetic traffic
- Jobs that entered
- SLI:
good_queue_delay_jobs / total_queued_jobs
3) Execution Success Rate (isis.execution-success-rate)#
- Good event:
- Terminal status
completed
- Terminal status
- Total event:
- Terminal statuses
completed | failed | timeout | provider_error - Excluding explicit user cancellations
- Terminal statuses
- SLI:
completed_jobs / terminal_jobs_excluding_user_cancel
4) End-to-End Completion (isis.e2e-completion-time)#
- Good event:
- Terminal status
completed completed_at - submitted_at <= 900s
- Terminal status
- Total event:
- Jobs accepted by
POST /api/v1/jobs - Excluding explicit user cancellations and synthetic traffic
- Jobs accepted by
- SLI:
good_e2e_jobs / total_accepted_jobs
Error Budgets (30 Days)#
| SLO ID | Target | Error Budget |
|---|---|---|
isis.submit-latency |
99.0% | 1.0% |
isis.queue-delay |
97.0% | 3.0% |
isis.execution-success-rate |
97.0% | 3.0% |
isis.e2e-completion-time |
95.0% | 5.0% |
Burn-Rate Alert Policy#
For each SLO:
- Fast-burn page:
- Burn rate >= 14 over 5 minutes
- Slow-burn ticket:
- Burn rate >= 2 over 1 hour
Recommended severity mapping:
SEV-1: Fast-burn onexecution-success-rateore2e-completion-timeSEV-2: Fast-burn onsubmit-latencyorqueue-delaySEV-3: Slow-burn on any SLO
Canonical Metric Contracts (for OBS-002 instrumentation)#
isis_generation_submit_latency_ms(histogram)- labels:
route,status_class,project_id
- labels:
isis_generation_queue_delay_seconds(histogram)- labels:
priority,workflow_type,project_id
- labels:
isis_generation_terminal_total(counter)- labels:
status,provider,workflow_type,project_id
- labels:
isis_generation_e2e_completion_seconds(histogram)- labels:
provider,workflow_type,project_id
- labels:
Release and Rollback Integration#
- Release gate evidence (
QG-006) must include current 7d/30d SLO compliance. - Auto-rollback tie-in:
- Submit-latency degradation threshold in
docs/domains/isis/remediation/ISIS_ROLLBACK_CRITERIA.md - Queue backlog and terminal failure regressions tied to SLO burn behavior.
- Submit-latency degradation threshold in
- On-call playbook integration:
docs/domains/isis/runbooks/creative-pipeline-oncall.mdscripts/isis/run_creative_pipeline_oncall_triage.sh
Traceability#
- Checklist:
OBS-001 - Next tasks:
OBS-002(metrics implementation)OBS-003(provider latency/failure dashboards)OBS-005(anomaly alerts)