Date: 2026-02-21
Scope: All Isis production rollouts
Sources: ISIS_TODOS.md, docs/domains/isis/remediation/ISIS_RISK_BOARD.md, docs/domains/isis/remediation/ISIS_MILESTONES.md
Purpose#
Define mandatory rollback criteria, authority, and verification requirements for every Isis production rollout surface.
Rollout Surfaces Covered#
- Generation API deployment (
apps/isis/generation-api) - Workflow registry deployment (
apps/isis/workflow-registry) - Output registry deployment (
apps/isis/output-registry) - GPU worker deployment (
apps/isis/gpu-worker) - RunPod endpoint image/config rollout (
docker/runpod/comfyui, RunPod endpoints) - Workflow/schema contract rollout (
libs/isis/workflows, registry schema, resolver contracts) - Database migration rollout (
libs/isis/database/prisma/schema.prismaand migrations) - Isis web app rollout (
apps/isis/web, once active in production) - SDK/API contract rollout (
@isis/client, OpenAPI/route contracts)
Severity and Rollback Decision Matrix#
| Severity | Trigger Class | Rollback Decision |
|---|---|---|
| P0 | Availability outage, data corruption risk, security incident, uncontrolled spend risk | Immediate rollback (no fix-forward window) |
| P1 | Major degradation to core workflow execution or provider routing | Rollback within 30 minutes unless verified hotfix is already in progress and ETA < 30 minutes |
| P2 | Partial degradation with bounded blast radius | Evaluate fix-forward vs rollback; default rollback if canary phase |
| P3 | Minor issue with workaround and no data/cost risk | Fix-forward preferred |
Global Auto-Rollback Criteria#
Rollback is mandatory when any condition is met during canary or first-hour post-promotion:
5xxerror rate > 2% for 10 consecutive minutes on any Isis core API.- Job terminal failure rate > 10% for 10 minutes relative to baseline.
- Submit-to-start latency p95 degrades by > 100% versus pre-release baseline.
- Queue backlog growth is monotonic for 15 minutes while worker throughput is flat.
- RunPod billing fields (
runtimeSeconds,costUsd) missing in > 0.5% of completed jobs. - Cost burn exceeds +20% forecast for 30 minutes without corresponding throughput gain.
- Any proven cross-tenant data access/control violation.
- Any detected production path returning simulated/placeholder output.
Rollout-Specific Rollback Criteria#
| Rollout Type | Rollback Triggers | Max Decision Time | Owner |
|---|---|---|---|
| Generation API | Submit failures, status route errors, queue state mismatch, auth regressions | 15 min | @oshun/isis-team |
| Workflow Registry | Validation mismatch, workflow publish failures, schema incompatibility | 30 min | @oshun/isis-team |
| Output Registry | Provenance/lineage write failures, retention/policy regressions, unauthorized access | 15 min | @oshun/isis-team |
| GPU Worker | Worker crash loops, execution timeout spikes, unsupported executor path activation | 15 min | @oshun/ml-team |
| RunPod Endpoint | Endpoint drift, failure-rate spike, routing mismatch, cost anomaly | 10 min | @oshun/devops |
| Schema/Contract | Registry-export to executor incompatibility, canonical validation failures | 30 min | @oshun/isis-team |
| DB Migration | Migration failure, lock contention, data integrity violation, rollback script failure | Immediate | @oshun/isis-team, @oshun/platform |
| Isis Web App | Core run-flow breakage, auth/session regression, broken output interaction paths | 30 min | @oshun/isis-team |
| SDK/API Contract | Client-breaking route mismatch, incompatible payload change without version gate | 30 min | @oshun/isis-team |
Authorization Matrix#
| Rollback Scope | Authorized Roles |
|---|---|
| Service deployment rollback | On-call engineer + service owner |
| RunPod endpoint rollback | On-call platform/devops + ML owner |
| Database migration rollback | DB owner + platform lead |
| Cross-service rollback (multi-surface) | Incident commander + @oshun/core-team delegate |
| Security-driven emergency rollback | Security lead + incident commander |
Canary and Promotion Rules#
- No full promotion without successful canary soak.
- Canary soak window must include:
- API error/latency metrics
- job lifecycle completion metrics
- RunPod cost/runtime metrics
- schema compatibility checks
- If rollback criteria trigger during canary:
- stop promotion immediately
- rollback to last known good revision/config
- open incident and block further promotion until root-cause disposition
Database Rollback Rules#
- Every production migration must have:
- pre-migration backup/snapshot
- explicit rollback path or compatible forward-fix path
- verification queries for data correctness
- Rollback is immediate when:
- migration fails partially
- data integrity constraints violate expected invariants
- write-path latency/lock contention exceeds safe thresholds
- If physical rollback is unsafe, execute controlled forward-fix plan with incident commander approval and written risk acceptance.
RunPod-Specific Rollback Rules#
- Roll back endpoint image/config when:
- capability map mismatch causes unsupported workflow routing
- failure rate exceeds threshold with correlated endpoint change
- cost anomaly criteria trigger after endpoint rollout
- Rollback action precedence:
- revert endpoint config/image
- revert routing map to previous version
- reduce canary traffic to zero
- fail back to approved secondary provider only if incident policy permits
Schema/Contract Rollback Rules#
- Roll back schema/contract release when:
- canonical validation blocks existing certified workflows
- registry exports fail executor preflight checks
- SDK/API compatibility tests fail for released clients
- Rollback mechanism:
- restore previous schema package version
- restore registry validation ruleset
- restore previous OpenAPI/SDK artifact set
Mandatory Rollback Checklist#
Before executing rollback:
- Confirm trigger and blast radius.
- Capture current metrics and logs for incident evidence.
- Identify last known good version/config.
- Notify incident channel and owning teams.
During rollback:
- Execute rollback plan for affected surface.
- Freeze further promotion/deploy activity.
- Monitor stabilization metrics continuously.
After rollback:
- Verify health/readiness/truthful dependency checks.
- Verify job submission, execution, and output retrieval on smoke flows.
- Verify cost/accounting fields remain complete.
- Publish incident update with rollback timestamp and impact summary.
- Create follow-up tasks before resuming rollout.
Exit Conditions After Rollback#
Rollout may resume only when all conditions are true:
- Triggering metric returns to baseline and remains stable for 30 minutes.
- No P0/P1 risk in
ISIS_RISK_BOARD.mdremains in triggered state. - Root-cause hypothesis is documented with owner and due date.
- Remediation tasks are created and prioritized in remediation backlog.
@oshun/core-teamapproves resumption for multi-surface incidents.
Traceability#
Primary checklist item:
WS00-006
Related rollout and operations items:
REL-001toREL-010F08-004toF08-008MIG-005OPS-001GATE-001toGATE-008