Eve library

Isis Production Rollback Criteria

Guide

Revision 1

On this page

Date: 2026-02-21
Scope: All Isis production rollouts
Sources: ISIS_TODOS.md, docs/domains/isis/remediation/ISIS_RISK_BOARD.md, docs/domains/isis/remediation/ISIS_MILESTONES.md

Purpose#

Define mandatory rollback criteria, authority, and verification requirements for every Isis production rollout surface.

Rollout Surfaces Covered#

  1. Generation API deployment (apps/isis/generation-api)
  2. Workflow registry deployment (apps/isis/workflow-registry)
  3. Output registry deployment (apps/isis/output-registry)
  4. GPU worker deployment (apps/isis/gpu-worker)
  5. RunPod endpoint image/config rollout (docker/runpod/comfyui, RunPod endpoints)
  6. Workflow/schema contract rollout (libs/isis/workflows, registry schema, resolver contracts)
  7. Database migration rollout (libs/isis/database/prisma/schema.prisma and migrations)
  8. Isis web app rollout (apps/isis/web, once active in production)
  9. SDK/API contract rollout (@isis/client, OpenAPI/route contracts)

Severity and Rollback Decision Matrix#

Severity Trigger Class Rollback Decision
P0 Availability outage, data corruption risk, security incident, uncontrolled spend risk Immediate rollback (no fix-forward window)
P1 Major degradation to core workflow execution or provider routing Rollback within 30 minutes unless verified hotfix is already in progress and ETA < 30 minutes
P2 Partial degradation with bounded blast radius Evaluate fix-forward vs rollback; default rollback if canary phase
P3 Minor issue with workaround and no data/cost risk Fix-forward preferred

Global Auto-Rollback Criteria#

Rollback is mandatory when any condition is met during canary or first-hour post-promotion:

  1. 5xx error rate > 2% for 10 consecutive minutes on any Isis core API.
  2. Job terminal failure rate > 10% for 10 minutes relative to baseline.
  3. Submit-to-start latency p95 degrades by > 100% versus pre-release baseline.
  4. Queue backlog growth is monotonic for 15 minutes while worker throughput is flat.
  5. RunPod billing fields (runtimeSeconds, costUsd) missing in > 0.5% of completed jobs.
  6. Cost burn exceeds +20% forecast for 30 minutes without corresponding throughput gain.
  7. Any proven cross-tenant data access/control violation.
  8. Any detected production path returning simulated/placeholder output.

Rollout-Specific Rollback Criteria#

Rollout Type Rollback Triggers Max Decision Time Owner
Generation API Submit failures, status route errors, queue state mismatch, auth regressions 15 min @oshun/isis-team
Workflow Registry Validation mismatch, workflow publish failures, schema incompatibility 30 min @oshun/isis-team
Output Registry Provenance/lineage write failures, retention/policy regressions, unauthorized access 15 min @oshun/isis-team
GPU Worker Worker crash loops, execution timeout spikes, unsupported executor path activation 15 min @oshun/ml-team
RunPod Endpoint Endpoint drift, failure-rate spike, routing mismatch, cost anomaly 10 min @oshun/devops
Schema/Contract Registry-export to executor incompatibility, canonical validation failures 30 min @oshun/isis-team
DB Migration Migration failure, lock contention, data integrity violation, rollback script failure Immediate @oshun/isis-team, @oshun/platform
Isis Web App Core run-flow breakage, auth/session regression, broken output interaction paths 30 min @oshun/isis-team
SDK/API Contract Client-breaking route mismatch, incompatible payload change without version gate 30 min @oshun/isis-team

Authorization Matrix#

Rollback Scope Authorized Roles
Service deployment rollback On-call engineer + service owner
RunPod endpoint rollback On-call platform/devops + ML owner
Database migration rollback DB owner + platform lead
Cross-service rollback (multi-surface) Incident commander + @oshun/core-team delegate
Security-driven emergency rollback Security lead + incident commander

Canary and Promotion Rules#

  1. No full promotion without successful canary soak.
  2. Canary soak window must include:
    • API error/latency metrics
    • job lifecycle completion metrics
    • RunPod cost/runtime metrics
    • schema compatibility checks
  3. If rollback criteria trigger during canary:
    • stop promotion immediately
    • rollback to last known good revision/config
    • open incident and block further promotion until root-cause disposition

Database Rollback Rules#

  1. Every production migration must have:
    • pre-migration backup/snapshot
    • explicit rollback path or compatible forward-fix path
    • verification queries for data correctness
  2. Rollback is immediate when:
    • migration fails partially
    • data integrity constraints violate expected invariants
    • write-path latency/lock contention exceeds safe thresholds
  3. If physical rollback is unsafe, execute controlled forward-fix plan with incident commander approval and written risk acceptance.

RunPod-Specific Rollback Rules#

  1. Roll back endpoint image/config when:
    • capability map mismatch causes unsupported workflow routing
    • failure rate exceeds threshold with correlated endpoint change
    • cost anomaly criteria trigger after endpoint rollout
  2. Rollback action precedence:
    • revert endpoint config/image
    • revert routing map to previous version
    • reduce canary traffic to zero
    • fail back to approved secondary provider only if incident policy permits

Schema/Contract Rollback Rules#

  1. Roll back schema/contract release when:
    • canonical validation blocks existing certified workflows
    • registry exports fail executor preflight checks
    • SDK/API compatibility tests fail for released clients
  2. Rollback mechanism:
    • restore previous schema package version
    • restore registry validation ruleset
    • restore previous OpenAPI/SDK artifact set

Mandatory Rollback Checklist#

Before executing rollback:

  • Confirm trigger and blast radius.
  • Capture current metrics and logs for incident evidence.
  • Identify last known good version/config.
  • Notify incident channel and owning teams.

During rollback:

  • Execute rollback plan for affected surface.
  • Freeze further promotion/deploy activity.
  • Monitor stabilization metrics continuously.

After rollback:

  • Verify health/readiness/truthful dependency checks.
  • Verify job submission, execution, and output retrieval on smoke flows.
  • Verify cost/accounting fields remain complete.
  • Publish incident update with rollback timestamp and impact summary.
  • Create follow-up tasks before resuming rollout.

Exit Conditions After Rollback#

Rollout may resume only when all conditions are true:

  1. Triggering metric returns to baseline and remains stable for 30 minutes.
  2. No P0/P1 risk in ISIS_RISK_BOARD.md remains in triggered state.
  3. Root-cause hypothesis is documented with owner and due date.
  4. Remediation tasks are created and prioritized in remediation backlog.
  5. @oshun/core-team approves resumption for multi-surface incidents.

Traceability#

Primary checklist item:

  • WS00-006

Related rollout and operations items:

  • REL-001 to REL-010
  • F08-004 to F08-008
  • MIG-005
  • OPS-001
  • GATE-001 to GATE-008