# Isis Creative Pipeline On-Call Playbook

Last Updated: 2026-02-24  
Scope: Creative generation pipeline incidents and SLO breaches across
`submit -> queue -> execute -> output`  
Primary Owners: `@oshun/isis-team`, `@oshun/devops`, `@oshun/platform`

## Purpose

Provide an operator-grade on-call playbook for incidents that degrade creative
pipeline reliability, throughput, or delivery SLOs.

This playbook complements:

- `docs/domains/isis/runbooks/runpod-oncall.md` (provider/runtime
  specialization)
- `docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md` (SLO contract)
- `docs/reference/runbooks/incident-response.md` (cross-platform incident
  process)

## Failure Classes

| Class ID | Incident Class              | Typical Signals                                           |
| -------- | --------------------------- | --------------------------------------------------------- | --------------------------- |
| `CP-001` | Service availability        | `/health` or `/ready` failures, job submit outages        |
| `CP-002` | Provider outage/degradation | `providers.healthy=false`, provider failure spikes        |
| `CP-003` | Queue saturation/starvation | saturation/starvation indicator `elevated                 | critical`, stuck-job growth |
| `CP-004` | SLO breach                  | fast/slow burn alerts, canary SLO/cost gate failure       |
| `CP-005` | Bad deploy regression       | post-deploy failure/latency jump, rollback recommendation |

## Severity Policy (Creative Pipeline)

- `SEV-1`
  - availability outage (`CP-001`) or
  - fast-burn breach on `isis.execution-success-rate` or
    `isis.e2e-completion-time`
- `SEV-2`
  - provider degradation (`CP-002`),
  - queue saturation (`CP-003`), or
  - fast-burn breach on `isis.submit-latency`/`isis.queue-delay`
- `SEV-3`
  - slow-burn on any pipeline SLO
- `SEV-4`
  - no active incident signal; monitoring-only follow-up

## First 15 Minutes

### 1) Run Triage Automation

```bash
ISIS_API_BASE_URL="https://<isis-api-host>" \
ISIS_ADMIN_TOKEN="..." \
ISIS_DATABASE_URL="postgres://..." \
STRICT=true \
scripts/isis/run_creative_pipeline_oncall_triage.sh
```

Outputs:

- `tmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.json`
- `tmp/isis-oncall/creative-pipeline-oncall-triage-<timestamp>.md`

### 2) Confirm Incident Class and Severity

Use triage report fields:

- `classification.severity`
- `classification.classes[]`
- `checks` failures for health/ready/providers/queue/SLO status

### 3) Open Incident Comms

Post in incident channel with:

- incident class (`CP-*`)
- current severity
- current mitigation action
- next update ETA (15 minutes)

## Playbook by Incident Class

### CP-001 Service Availability

1. Validate `/health` and `/ready` from triage artifacts.
2. Roll back latest deployment/config change if failure began post-release.
3. Confirm submit path recovery (`POST /api/v1/jobs` success trend).
4. Keep canary rollout frozen until stability is sustained.

### CP-002 Provider Outage/Degradation

1. Run provider outage drill playbook procedures:
   - `docs/domains/isis/runbooks/provider-outage-drill.md`
2. Validate provider operations:
   - `/api/v1/jobs/providers/operations`
3. If needed, force fallback routing and hold ramp progression.
4. Verify synthetic jobs progress in non-failed states.

### CP-003 Queue Saturation/Starvation

1. Inspect queue indicators from triage:
   - `.queue.indicators.saturation`
   - `.queue.indicators.starvation`
2. Pause canary promotion and reduce canary percentage when pressure is rising.
3. Reconcile endpoint drift and check worker capacity allocation.
4. Replay dead-letter only after root cause is contained.

### CP-004 SLO Breach

1. Run canary verifier:

```bash
ISIS_DATABASE_URL="postgres://..." \
FAIL_ON_BREACH=false \
scripts/isis/verify_canary_slo_cost.sh
```

2. Map failed checks to SLO IDs:
   - submit latency -> `isis.submit-latency`
   - queue delay -> `isis.queue-delay`
   - execution success -> `isis.execution-success-rate`
   - e2e completion -> `isis.e2e-completion-time`
3. Apply burn-rate severity mapping from
   `docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md`.
4. Trigger rollback path if fast-burn criteria are met.

### CP-005 Bad Deploy Regression

1. Generate rollback decision artifact:

```bash
CURRENT_PERCENTAGE=25 \
AUTO_ROLLBACK_ENABLED=true \
FAIL_ON_ROLLBACK=false \
scripts/isis/advance_queue_canary_rollout.sh
```

2. If action is `rollback`, set target rollout to `0%` and execute rollback
   procedure.
3. Validate post-rollback stabilization before any re-promotion.

## SLO Breach Escalation Matrix

| Breach Type                    | Immediate Owner    | Escalate To                         |
| ------------------------------ | ------------------ | ----------------------------------- |
| Submit/queue latency fast burn | `@oshun/devops`    | `@oshun/platform`                   |
| Execution/e2e fast burn        | `@oshun/isis-team` | `@oshun/platform`, `@oshun/ml-team` |
| Slow-burn persistent drift     | `@oshun/isis-team` | `@oshun/core-team`                  |
| Cost + SLO coupled degradation | `@oshun/devops`    | `@oshun/platform`, `@oshun/ml-team` |

## Exit Criteria

Incident may close only when:

1. Required triage checks are passing or approved exceptions are documented.
2. SLO breach condition has stopped burning at incident threshold.
3. Queue/provider health has stabilized for an agreed observation window.
4. Follow-up action items are filed with owners and due dates.
5. Post-incident review is created for `SEV-1`/`SEV-2`.

## Related Artifacts

- `scripts/isis/run_creative_pipeline_oncall_triage.sh`
- `docs/domains/isis/runbooks/runpod-oncall.md`
- `docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md`
- `docs/domains/isis/remediation/ISIS_CANARY_SLO_COST_VERIFICATION.md`
- `docs/domains/isis/remediation/ISIS_POST_INCIDENT_REVIEW_TEMPLATE.md`
- `docs/reference/runbooks/incident-response.md`
