# Post-Incident Review Process

**Purpose**: Learn from incidents to improve system reliability, response
processes, and prevent recurrence

## PIR Overview and Philosophy

### Blameless Post-Mortems

- **Focus on systems and processes**, not individual actions
- **Assume good intentions** from all team members
- **Encourage transparency** to maximize learning
- **Multiple contributing factors** rather than single root cause
- **Systemic improvements** over individual accountability

### PIR Objectives

1. **Understand what happened** with complete timeline and context
2. **Identify contributing factors** both technical and organizational
3. **Extract learnings** applicable to future incidents
4. **Define action items** to prevent recurrence and improve response
5. **Share knowledge** across the engineering organization

## PIR Triggers and Timeline

### When PIR is Required

- **All P0 incidents** (within 72 hours)
- **P1 incidents lasting >4 hours** (within 1 week)
- **Security incidents** (within 48 hours)
- **Customer data impact** (within 72 hours)
- **Novel failure modes** (any severity, within 1 week)

### PIR Timeline

```
Incident Resolution → 24 hours → PIR Scheduled
Incident Resolution → 72 hours → PIR Data Collection Complete
Incident Resolution → 1 week → PIR Meeting Conducted
PIR Meeting → 1 week → PIR Report Published
PIR Report → 2 weeks → Action Items Assigned
PIR Report → 1 month → Action Items Progress Review
```

## PIR Roles and Responsibilities

### PIR Owner (Incident Commander or delegate)

- **Schedule PIR meeting** within appropriate timeline
- **Collect incident data** and prepare timeline
- **Facilitate PIR discussion** keeping focus on learnings
- **Write PIR report** summarizing findings and actions
- **Track action item** completion and effectiveness

### PIR Participants

- **Incident responders** (on-call engineer, technical leads, SMEs)
- **Service owners** for affected systems
- **Engineering manager** for affected teams
- **Product manager** (if customer impact)
- **Security representative** (for security incidents)
- **Site reliability engineer** (observability perspective)

### PIR Meeting Facilitator (if different from PIR Owner)

- **Guide discussion** through structured agenda
- **Encourage participation** from all attendees
- **Keep focus** on systemic issues rather than individual actions
- **Manage time** to cover all important areas
- **Capture insights** and action items during discussion

## PIR Data Collection

### Information Gathering Checklist

- [ ] Complete incident timeline with timestamps
- [ ] All relevant logs and error messages
- [ ] Monitoring data and graphs covering incident period
- [ ] Chat logs from incident coordination
- [ ] List of all people who participated in response
- [ ] Customer impact data and support ticket volume
- [ ] Business impact assessment (revenue, SLA breaches)
- [ ] External vendor communications if applicable

### Timeline Construction

```
PIR Timeline Template:

[TIME] - Detection
- How was incident first detected?
- What monitoring/alerting fired?
- Who was the first responder?

[TIME] - Investigation Start
- What were the initial symptoms observed?
- What hypotheses were formed?
- What investigation steps were taken?

[TIME] - Key Findings
- When were significant clues discovered?
- How did understanding evolve?
- What false leads were pursued?

[TIME] - Mitigation Attempts
- What fixes were attempted?
- Which worked/didn't work and why?
- Any unintended consequences?

[TIME] - Resolution
- What ultimately resolved the incident?
- How was resolution verified?
- When was normal service confirmed?

[TIME] - Post-Resolution
- Any follow-up issues discovered?
- Customer communication completed?
- Monitoring returned to normal?
```

### Data Preservation

```bash
# Preserve logs for PIR analysis
kubectl logs -l app=[service] --since=[incident-start] > incident-[id]-[service].log

# Export monitoring data
grafana-cli dashboards export [dashboard-id] --time-range=[incident-period]

# Preserve incident communication
slack-export #lilith-incidents --from=[start-time] --to=[end-time]

# Database query logs (if applicable)
kubectl exec postgres-primary-0 -- pg_dump --schema-only > pre-incident-schema.sql
```

## PIR Meeting Structure

### Pre-Meeting Preparation (1 week before)

- [ ] Schedule 90-minute meeting with all key participants
- [ ] Distribute incident timeline and background materials
- [ ] Ask participants to review materials beforehand
- [ ] Prepare specific questions for each participant
- [ ] Set up collaborative document for note-taking

### Meeting Agenda (90 minutes)

#### Opening (10 minutes)

- **PIR philosophy reminder** - blameless, learning-focused
- **Incident summary** - high-level context for all attendees
- **Meeting objectives** - what we want to accomplish
- **Participant introductions** - role in incident response

#### Timeline Review (20 minutes)

- **Walk through timeline** chronologically
- **Validate accuracy** with incident responders
- **Fill in gaps** where timeline incomplete
- **Clarify decision points** and reasoning

#### Deep Dive Analysis (40 minutes)

- **Root cause analysis** - what caused the incident?
- **Contributing factors** - what made it worse or harder to resolve?
- **Response effectiveness** - what went well/poorly in response?
- **Communication review** - internal and external communication quality

#### Learnings and Actions (15 minutes)

- **Key insights** - what did we learn?
- **Preventive actions** - how to prevent recurrence?
- **Response improvements** - how to respond better next time?
- **Monitoring enhancements** - how to detect earlier/better?

#### Wrap-up (5 minutes)

- **Action item assignment** with owners and timelines
- **PIR report timeline** commitment
- **Follow-up meeting** if needed

### PIR Discussion Questions

#### Understanding the Incident

- What was the sequence of events that led to this incident?
- Were there any warning signs that we missed?
- How did the incident progress and evolve?
- What made this incident particularly challenging?

#### Response Analysis

- How effective was our detection and alerting?
- Did we escalate appropriately and at the right time?
- Were we missing any critical information during response?
- Did we have the right people engaged at the right time?

#### Contributing Factors

- What systemic issues contributed to this incident?
- Were there process gaps that made response more difficult?
- Did tool limitations impact our ability to respond?
- Were there knowledge gaps that slowed resolution?

#### Learning Opportunities

- What surprised us about this incident?
- What worked particularly well that we should reinforce?
- What would we do differently if faced with this again?
- How can we share these learnings with other teams?

## PIR Report Structure

### Executive Summary

```
Incident: [Brief description]
Duration: [Start time] to [resolution time] ([total duration])
Impact: [User impact, business impact, affected services]
Root Cause: [Primary cause with brief explanation]
Resolution: [How incident was resolved]
Key Learnings: [Top 3 insights from incident]
Action Items: [Number of actions with completion timeline]
```

### Detailed Timeline

```
Complete chronological sequence of events from detection through resolution,
including:
- Decision points and rationale
- Investigation paths taken
- Mitigation attempts
- Communication milestones
- External factors or dependencies
```

### Impact Analysis

```
User Impact:
- Number of users affected: [specific numbers]
- Duration of impact: [timeline per service]
- Functionality affected: [detailed list]
- Geographic distribution: [regions affected]

Business Impact:
- Revenue impact: [quantified if possible]
- SLA breaches: [which SLAs and duration]
- Customer support volume: [ticket increases]
- Reputation/PR impact: [if applicable]

Technical Impact:
- Services affected: [complete list]
- Data integrity: [any data issues]
- Security implications: [if applicable]
- Performance degradation: [metrics]
```

### Root Cause Analysis

```
Primary Root Cause:
[Detailed explanation of the fundamental issue that caused the incident]

Contributing Factors:
1. [Factor 1 with explanation]
2. [Factor 2 with explanation]
3. [Factor 3 with explanation]

Why These Factors Existed:
[Analysis of why the contributing factors were present in the system]

Detection and Response Factors:
[What made the incident harder to detect or resolve]
```

### What Went Well

```
- [Positive aspects of incident response]
- [Effective tools or processes]
- [Good decision-making examples]
- [Effective communication examples]
- [Quick mitigation successes]
```

### What Could Be Improved

```
- [Areas where response could be better]
- [Process gaps identified]
- [Tool limitations encountered]
- [Communication breakdowns]
- [Knowledge gaps revealed]
```

### Action Items

#### Prevention Actions

```
Action: [Specific action to prevent recurrence]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
```

#### Detection/Response Improvements

```
Action: [Specific action to improve response]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
```

#### Process/Tool Enhancements

```
Action: [Specific process or tool improvement]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
```

## PIR Follow-up Process

### Action Item Tracking

- **Weekly review** of action item progress by PIR owner
- **Bi-weekly updates** to stakeholders on progress
- **Monthly review** of effectiveness of completed actions
- **Quarterly analysis** of trends across multiple PIRs

### Action Item Categories

```
Immediate (1-2 weeks):
- Critical fixes to prevent immediate recurrence
- Hotfixes for dangerous conditions
- Documentation updates for responders

Short-term (1-2 months):
- Process improvements
- Tool enhancements
- Training additions
- Monitoring improvements

Long-term (3-6 months):
- Architecture changes
- System redesigns
- Tool replacements
- Organizational changes
```

### PIR Report Distribution

```
Required Recipients:
- All incident participants
- Engineering leadership
- Product management (if customer impact)
- Security team (for security incidents)
- Customer support leadership

Optional Recipients:
- Broader engineering teams (for learning)
- Executive team (for P0 incidents)
- Customer success team
- Technical writing team
```

## PIR Quality and Effectiveness

### PIR Success Metrics

- **Action item completion rate** (target: >90% within timeline)
- **Recurrence prevention** (same root cause incidents)
- **Response time improvement** (faster future incident response)
- **Learning effectiveness** (knowledge sharing across teams)

### PIR Review Criteria

- [ ] Timeline is complete and accurate
- [ ] Root cause analysis is thorough and systemic
- [ ] Action items are specific and actionable
- [ ] Learnings are clearly articulated
- [ ] Report is well-written and clear
- [ ] Blameless tone maintained throughout

### Common PIR Pitfalls

#### Blame-focused Analysis

- Focusing on individual mistakes rather than system failures
- Suggesting individual training as primary solution
- Using judgmental language about decisions made under pressure

#### Shallow Analysis

- Stopping at proximate cause rather than digging deeper
- Missing organizational and process contributing factors
- Not considering why good people made reasonable decisions that led to problems

#### Weak Action Items

- Vague actions that can't be measured
- No assigned owners or timelines
- Actions that don't address root causes
- Too many low-priority actions diluting focus

### PIR Improvement Process

- **Monthly PIR quality review** by engineering leadership
- **Quarterly trend analysis** across all PIRs
- **Annual PIR process review** and updates
- **Training updates** based on PIR learnings

## PIR Templates and Tools

### PIR Meeting Invite Template

```
Subject: Post-Incident Review - [Incident Description] - [Date]

We'll be conducting a post-incident review for the [incident description] that occurred on [date].

When: [Date/Time]
Duration: 90 minutes
Where: [Meeting room/video link]

Please review the incident timeline [link] before the meeting.

Agenda:
- Timeline review and validation
- Root cause analysis
- Response effectiveness discussion
- Action item identification

Remember: This is a blameless review focused on learning and improvement.
```

### PIR Report Template

[Complete template available in separate document: `pir-report-template.md`]

### Action Item Tracking Spreadsheet

```
Columns: Action ID, Description, Owner, Priority, Due Date, Status, Notes, Completion Date
```

---

**Remember**: The goal of PIR is learning and improvement, not blame. Focus on
systemic issues and be honest about what happened to maximize learning for the
entire organization.
