# Isis Production Rollback Criteria

Date: 2026-02-21  
Scope: All Isis production rollouts  
Sources: `ISIS_TODOS.md`, `docs/domains/isis/remediation/ISIS_RISK_BOARD.md`, `docs/domains/isis/remediation/ISIS_MILESTONES.md`

## Purpose

Define mandatory rollback criteria, authority, and verification requirements for every Isis production rollout surface.

## Rollout Surfaces Covered

1. Generation API deployment (`apps/isis/generation-api`)
2. Workflow registry deployment (`apps/isis/workflow-registry`)
3. Output registry deployment (`apps/isis/output-registry`)
4. GPU worker deployment (`apps/isis/gpu-worker`)
5. RunPod endpoint image/config rollout (`docker/runpod/comfyui`, RunPod endpoints)
6. Workflow/schema contract rollout (`libs/isis/workflows`, registry schema, resolver contracts)
7. Database migration rollout (`libs/isis/database/prisma/schema.prisma` and migrations)
8. Isis web app rollout (`apps/isis/web`, once active in production)
9. SDK/API contract rollout (`@isis/client`, OpenAPI/route contracts)

## Severity and Rollback Decision Matrix

| Severity | Trigger Class | Rollback Decision |
|---|---|---|
| P0 | Availability outage, data corruption risk, security incident, uncontrolled spend risk | Immediate rollback (no fix-forward window) |
| P1 | Major degradation to core workflow execution or provider routing | Rollback within 30 minutes unless verified hotfix is already in progress and ETA < 30 minutes |
| P2 | Partial degradation with bounded blast radius | Evaluate fix-forward vs rollback; default rollback if canary phase |
| P3 | Minor issue with workaround and no data/cost risk | Fix-forward preferred |

## Global Auto-Rollback Criteria

Rollback is mandatory when any condition is met during canary or first-hour post-promotion:

1. `5xx` error rate > 2% for 10 consecutive minutes on any Isis core API.
2. Job terminal failure rate > 10% for 10 minutes relative to baseline.
3. Submit-to-start latency p95 degrades by > 100% versus pre-release baseline.
4. Queue backlog growth is monotonic for 15 minutes while worker throughput is flat.
5. RunPod billing fields (`runtimeSeconds`, `costUsd`) missing in > 0.5% of completed jobs.
6. Cost burn exceeds +20% forecast for 30 minutes without corresponding throughput gain.
7. Any proven cross-tenant data access/control violation.
8. Any detected production path returning simulated/placeholder output.

## Rollout-Specific Rollback Criteria

| Rollout Type | Rollback Triggers | Max Decision Time | Owner |
|---|---|---|---|
| Generation API | Submit failures, status route errors, queue state mismatch, auth regressions | 15 min | `@oshun/isis-team` |
| Workflow Registry | Validation mismatch, workflow publish failures, schema incompatibility | 30 min | `@oshun/isis-team` |
| Output Registry | Provenance/lineage write failures, retention/policy regressions, unauthorized access | 15 min | `@oshun/isis-team` |
| GPU Worker | Worker crash loops, execution timeout spikes, unsupported executor path activation | 15 min | `@oshun/ml-team` |
| RunPod Endpoint | Endpoint drift, failure-rate spike, routing mismatch, cost anomaly | 10 min | `@oshun/devops` |
| Schema/Contract | Registry-export to executor incompatibility, canonical validation failures | 30 min | `@oshun/isis-team` |
| DB Migration | Migration failure, lock contention, data integrity violation, rollback script failure | Immediate | `@oshun/isis-team`, `@oshun/platform` |
| Isis Web App | Core run-flow breakage, auth/session regression, broken output interaction paths | 30 min | `@oshun/isis-team` |
| SDK/API Contract | Client-breaking route mismatch, incompatible payload change without version gate | 30 min | `@oshun/isis-team` |

## Authorization Matrix

| Rollback Scope | Authorized Roles |
|---|---|
| Service deployment rollback | On-call engineer + service owner |
| RunPod endpoint rollback | On-call platform/devops + ML owner |
| Database migration rollback | DB owner + platform lead |
| Cross-service rollback (multi-surface) | Incident commander + `@oshun/core-team` delegate |
| Security-driven emergency rollback | Security lead + incident commander |

## Canary and Promotion Rules

1. No full promotion without successful canary soak.
2. Canary soak window must include:
   - API error/latency metrics
   - job lifecycle completion metrics
   - RunPod cost/runtime metrics
   - schema compatibility checks
3. If rollback criteria trigger during canary:
   - stop promotion immediately
   - rollback to last known good revision/config
   - open incident and block further promotion until root-cause disposition

## Database Rollback Rules

1. Every production migration must have:
   - pre-migration backup/snapshot
   - explicit rollback path or compatible forward-fix path
   - verification queries for data correctness
2. Rollback is immediate when:
   - migration fails partially
   - data integrity constraints violate expected invariants
   - write-path latency/lock contention exceeds safe thresholds
3. If physical rollback is unsafe, execute controlled forward-fix plan with incident commander approval and written risk acceptance.

## RunPod-Specific Rollback Rules

1. Roll back endpoint image/config when:
   - capability map mismatch causes unsupported workflow routing
   - failure rate exceeds threshold with correlated endpoint change
   - cost anomaly criteria trigger after endpoint rollout
2. Rollback action precedence:
   - revert endpoint config/image
   - revert routing map to previous version
   - reduce canary traffic to zero
   - fail back to approved secondary provider only if incident policy permits

## Schema/Contract Rollback Rules

1. Roll back schema/contract release when:
   - canonical validation blocks existing certified workflows
   - registry exports fail executor preflight checks
   - SDK/API compatibility tests fail for released clients
2. Rollback mechanism:
   - restore previous schema package version
   - restore registry validation ruleset
   - restore previous OpenAPI/SDK artifact set

## Mandatory Rollback Checklist

Before executing rollback:
- Confirm trigger and blast radius.
- Capture current metrics and logs for incident evidence.
- Identify last known good version/config.
- Notify incident channel and owning teams.

During rollback:
- Execute rollback plan for affected surface.
- Freeze further promotion/deploy activity.
- Monitor stabilization metrics continuously.

After rollback:
- Verify health/readiness/truthful dependency checks.
- Verify job submission, execution, and output retrieval on smoke flows.
- Verify cost/accounting fields remain complete.
- Publish incident update with rollback timestamp and impact summary.
- Create follow-up tasks before resuming rollout.

## Exit Conditions After Rollback

Rollout may resume only when all conditions are true:

1. Triggering metric returns to baseline and remains stable for 30 minutes.
2. No P0/P1 risk in `ISIS_RISK_BOARD.md` remains in triggered state.
3. Root-cause hypothesis is documented with owner and due date.
4. Remediation tasks are created and prioritized in remediation backlog.
5. `@oshun/core-team` approves resumption for multi-surface incidents.

## Traceability

Primary checklist item:
- `WS00-006`

Related rollout and operations items:
- `REL-001` to `REL-010`
- `F08-004` to `F08-008`
- `MIG-005`
- `OPS-001`
- `GATE-001` to `GATE-008`
