Eve library
Isis Post-Incident Review Template (Generation Failures and Cost Overruns)
Guide
Revision 1
On this page
Date: YYYY-MM-DD
Incident ID: <incident-id>
Severity: SEV-1 / SEV-2 / SEV-3
Template Owner: @oshun/devops
- Incident commander:
- Scribe:
- Domain owners involved:
- Environment(s):
- Start time (UTC):
- End time (UTC):
- Time to detect (MTTD):
- Time to mitigate (MTTM):
- Time to resolve (MTTR):
2. Executive Summary#
- What happened:
- Why it mattered:
- User/business impact:
- Current status:
3. Impact Assessment#
3.1 Generation Reliability Impact#
| Metric |
Baseline |
Incident Peak |
Recovery Value |
Breach Duration |
| Submit success rate |
|
|
|
|
| Queue delay p95 |
|
|
|
|
| Execution success rate |
|
|
|
|
| End-to-end completion SLI |
|
|
|
|
| Failed jobs count |
|
|
|
|
3.2 Cost/Budget Impact#
| Metric |
Forecast |
Actual During Incident |
Variance |
Notes |
| RunPod spend (USD/hour) |
|
|
|
|
| RunPod monthly estimate (USD) |
|
|
|
|
| Cost per runtime second (USD) |
|
|
|
|
| Canary cost regression % |
|
|
|
|
| Budget guardrail mode transitions |
|
|
|
|
4. Customer and Internal Blast Radius#
- Affected projects/workspaces:
- Affected workflows/classes (
3d, avatar, video, etc.):
- Number of users/teams impacted:
- Data integrity impact (if any):
- Regulatory/compliance impact (if any):
5. Detection and Alerting Effectiveness#
| Signal / Alert |
Triggered (Y/N) |
Trigger Time |
Expected? |
Gap |
| Provider failure-rate anomaly |
|
|
|
|
| Queue growth anomaly |
|
|
|
|
| RunPod cost spike anomaly |
|
|
|
|
| Guardrail throttle/block state |
|
|
|
|
| Synthetic drill / canary gate checks |
|
|
|
|
6. Detailed Timeline (UTC)#
| Time |
Event |
Owner |
Evidence |
|
Incident started |
|
|
|
First mitigation applied |
|
|
|
Failover / rollback decision |
|
|
|
Recovery validated |
|
|
|
Incident resolved |
|
|
7. Technical Root Cause Analysis#
7.1 Failure Path Breakdown#
- Submit path:
- Queue/dispatch path:
- Provider execution path (RunPod/RunComfy):
- Worker execution path:
- Output/provenance/model storage path:
7.2 Root Cause Classification#
- Primary root cause:
- Contributing factors:
- Why safeguards did not prevent impact:
- Why detection/response speed was insufficient (if applicable):
7.3 Five Whys#
- Why:
- Why:
- Why:
- Why:
- Why:
8. Cost Overrun Analysis (Required for spend incidents)#
- Forecast model used:
- Driver(s) of overrun (retry storms, canary misconfig, worker overprovisioning,
etc.):
- Cost controls in place at incident start:
- Which controls failed or were missing:
- Guardrail behavior observed (soft/hard limit transitions):
- Cost recovered/avoided after mitigation:
| Remediation |
Type (mitigation/fix) |
Owner |
Completed At |
|
|
|
|
|
|
|
|
10. Corrective and Preventive Actions (CAPA)#
| Action ID |
Description |
Category (code/config/process/ops) |
Owner |
Due Date |
Tracking Ticket |
| CAPA-001 |
|
|
|
|
|
| CAPA-002 |
|
|
|
|
|
| CAPA-003 |
|
|
|
|
|
11. Validation Plan for CAPA#
- How fixes will be tested:
- Rollout strategy:
- Rollback strategy:
- Success criteria for closure:
12. Documentation and Runbook Updates#
- Runbooks updated:
- Dashboards/alerts updated:
- Configuration defaults updated:
- Training/on-call playbook updates:
13. Sign-Off#
- Incident commander sign-off:
- Domain owner sign-off:
- SRE sign-off:
- Security/compliance sign-off (if required):
- Date closed:
14. Required Evidence Attachments#
- Incident timeline export.
GET /api/v1/jobs/providers/operations and /api/v1/jobs/runpod/operations
snapshots.
- Relevant guardrail/canary verification reports.
- Cost breakdown artifact(s) and SQL/report queries used.
- Links to merged remediation PRs and tracking tickets.