Eve library

Isis Post-Incident Review Template (Generation Failures and Cost Overruns)

Guide

Revision 1

On this page

Date: YYYY-MM-DD
Incident ID: <incident-id>
Severity: SEV-1 / SEV-2 / SEV-3
Template Owner: @oshun/devops

1. Incident Metadata#

  • Incident commander:
  • Scribe:
  • Domain owners involved:
  • Environment(s):
  • Start time (UTC):
  • End time (UTC):
  • Time to detect (MTTD):
  • Time to mitigate (MTTM):
  • Time to resolve (MTTR):

2. Executive Summary#

  • What happened:
  • Why it mattered:
  • User/business impact:
  • Current status:

3. Impact Assessment#

3.1 Generation Reliability Impact#

Metric Baseline Incident Peak Recovery Value Breach Duration
Submit success rate
Queue delay p95
Execution success rate
End-to-end completion SLI
Failed jobs count

3.2 Cost/Budget Impact#

Metric Forecast Actual During Incident Variance Notes
RunPod spend (USD/hour)
RunPod monthly estimate (USD)
Cost per runtime second (USD)
Canary cost regression %
Budget guardrail mode transitions

4. Customer and Internal Blast Radius#

  • Affected projects/workspaces:
  • Affected workflows/classes (3d, avatar, video, etc.):
  • Number of users/teams impacted:
  • Data integrity impact (if any):
  • Regulatory/compliance impact (if any):

5. Detection and Alerting Effectiveness#

Signal / Alert Triggered (Y/N) Trigger Time Expected? Gap
Provider failure-rate anomaly
Queue growth anomaly
RunPod cost spike anomaly
Guardrail throttle/block state
Synthetic drill / canary gate checks

6. Detailed Timeline (UTC)#

Time Event Owner Evidence
Incident started
First mitigation applied
Failover / rollback decision
Recovery validated
Incident resolved

7. Technical Root Cause Analysis#

7.1 Failure Path Breakdown#

  • Submit path:
  • Queue/dispatch path:
  • Provider execution path (RunPod/RunComfy):
  • Worker execution path:
  • Output/provenance/model storage path:

7.2 Root Cause Classification#

  • Primary root cause:
  • Contributing factors:
  • Why safeguards did not prevent impact:
  • Why detection/response speed was insufficient (if applicable):

7.3 Five Whys#

  1. Why:
  2. Why:
  3. Why:
  4. Why:
  5. Why:

8. Cost Overrun Analysis (Required for spend incidents)#

  • Forecast model used:
  • Driver(s) of overrun (retry storms, canary misconfig, worker overprovisioning, etc.):
  • Cost controls in place at incident start:
  • Which controls failed or were missing:
  • Guardrail behavior observed (soft/hard limit transitions):
  • Cost recovered/avoided after mitigation:

9. Immediate Remediations Executed#

Remediation Type (mitigation/fix) Owner Completed At

10. Corrective and Preventive Actions (CAPA)#

Action ID Description Category (code/config/process/ops) Owner Due Date Tracking Ticket
CAPA-001
CAPA-002
CAPA-003

11. Validation Plan for CAPA#

  • How fixes will be tested:
  • Rollout strategy:
  • Rollback strategy:
  • Success criteria for closure:

12. Documentation and Runbook Updates#

  • Runbooks updated:
  • Dashboards/alerts updated:
  • Configuration defaults updated:
  • Training/on-call playbook updates:

13. Sign-Off#

  • Incident commander sign-off:
  • Domain owner sign-off:
  • SRE sign-off:
  • Security/compliance sign-off (if required):
  • Date closed:

14. Required Evidence Attachments#

  • Incident timeline export.
  • GET /api/v1/jobs/providers/operations and /api/v1/jobs/runpod/operations snapshots.
  • Relevant guardrail/canary verification reports.
  • Cost breakdown artifact(s) and SQL/report queries used.
  • Links to merged remediation PRs and tracking tickets.