# Isis Post-Incident Review Template (Generation Failures and Cost Overruns)

Date: YYYY-MM-DD  
Incident ID: `<incident-id>`  
Severity: `SEV-1` / `SEV-2` / `SEV-3`  
Template Owner: `@oshun/devops`

## 1. Incident Metadata

- Incident commander:
- Scribe:
- Domain owners involved:
- Environment(s):
- Start time (UTC):
- End time (UTC):
- Time to detect (MTTD):
- Time to mitigate (MTTM):
- Time to resolve (MTTR):

## 2. Executive Summary

- What happened:
- Why it mattered:
- User/business impact:
- Current status:

## 3. Impact Assessment

### 3.1 Generation Reliability Impact

| Metric                    | Baseline | Incident Peak | Recovery Value | Breach Duration |
| ------------------------- | -------- | ------------- | -------------- | --------------- |
| Submit success rate       |          |               |                |                 |
| Queue delay p95           |          |               |                |                 |
| Execution success rate    |          |               |                |                 |
| End-to-end completion SLI |          |               |                |                 |
| Failed jobs count         |          |               |                |                 |

### 3.2 Cost/Budget Impact

| Metric                            | Forecast | Actual During Incident | Variance | Notes |
| --------------------------------- | -------- | ---------------------- | -------- | ----- |
| RunPod spend (USD/hour)           |          |                        |          |       |
| RunPod monthly estimate (USD)     |          |                        |          |       |
| Cost per runtime second (USD)     |          |                        |          |       |
| Canary cost regression %          |          |                        |          |       |
| Budget guardrail mode transitions |          |                        |          |       |

## 4. Customer and Internal Blast Radius

- Affected projects/workspaces:
- Affected workflows/classes (`3d`, `avatar`, `video`, etc.):
- Number of users/teams impacted:
- Data integrity impact (if any):
- Regulatory/compliance impact (if any):

## 5. Detection and Alerting Effectiveness

| Signal / Alert                       | Triggered (Y/N) | Trigger Time | Expected? | Gap |
| ------------------------------------ | --------------- | ------------ | --------- | --- |
| Provider failure-rate anomaly        |                 |              |           |     |
| Queue growth anomaly                 |                 |              |           |     |
| RunPod cost spike anomaly            |                 |              |           |     |
| Guardrail throttle/block state       |                 |              |           |     |
| Synthetic drill / canary gate checks |                 |              |           |     |

## 6. Detailed Timeline (UTC)

| Time | Event                        | Owner | Evidence |
| ---- | ---------------------------- | ----- | -------- |
|      | Incident started             |       |          |
|      | First mitigation applied     |       |          |
|      | Failover / rollback decision |       |          |
|      | Recovery validated           |       |          |
|      | Incident resolved            |       |          |

## 7. Technical Root Cause Analysis

### 7.1 Failure Path Breakdown

- Submit path:
- Queue/dispatch path:
- Provider execution path (RunPod/RunComfy):
- Worker execution path:
- Output/provenance/model storage path:

### 7.2 Root Cause Classification

- Primary root cause:
- Contributing factors:
- Why safeguards did not prevent impact:
- Why detection/response speed was insufficient (if applicable):

### 7.3 Five Whys

1. Why:
2. Why:
3. Why:
4. Why:
5. Why:

## 8. Cost Overrun Analysis (Required for spend incidents)

- Forecast model used:
- Driver(s) of overrun (retry storms, canary misconfig, worker overprovisioning,
  etc.):
- Cost controls in place at incident start:
- Which controls failed or were missing:
- Guardrail behavior observed (soft/hard limit transitions):
- Cost recovered/avoided after mitigation:

## 9. Immediate Remediations Executed

| Remediation | Type (mitigation/fix) | Owner | Completed At |
| ----------- | --------------------- | ----- | ------------ |
|             |                       |       |              |
|             |                       |       |              |

## 10. Corrective and Preventive Actions (CAPA)

| Action ID | Description | Category (code/config/process/ops) | Owner | Due Date | Tracking Ticket |
| --------- | ----------- | ---------------------------------- | ----- | -------- | --------------- |
| CAPA-001  |             |                                    |       |          |                 |
| CAPA-002  |             |                                    |       |          |                 |
| CAPA-003  |             |                                    |       |          |                 |

## 11. Validation Plan for CAPA

- How fixes will be tested:
- Rollout strategy:
- Rollback strategy:
- Success criteria for closure:

## 12. Documentation and Runbook Updates

- Runbooks updated:
- Dashboards/alerts updated:
- Configuration defaults updated:
- Training/on-call playbook updates:

## 13. Sign-Off

- Incident commander sign-off:
- Domain owner sign-off:
- SRE sign-off:
- Security/compliance sign-off (if required):
- Date closed:

## 14. Required Evidence Attachments

- Incident timeline export.
- `GET /api/v1/jobs/providers/operations` and `/api/v1/jobs/runpod/operations`
  snapshots.
- Relevant guardrail/canary verification reports.
- Cost breakdown artifact(s) and SQL/report queries used.
- Links to merged remediation PRs and tracking tickets.
