Eve library

Post-Incident Review Process

Runbook

Revision 1

On this page

Purpose: Learn from incidents to improve system reliability, response processes, and prevent recurrence

PIR Overview and Philosophy#

Blameless Post-Mortems#

  • Focus on systems and processes, not individual actions
  • Assume good intentions from all team members
  • Encourage transparency to maximize learning
  • Multiple contributing factors rather than single root cause
  • Systemic improvements over individual accountability

PIR Objectives#

  1. Understand what happened with complete timeline and context
  2. Identify contributing factors both technical and organizational
  3. Extract learnings applicable to future incidents
  4. Define action items to prevent recurrence and improve response
  5. Share knowledge across the engineering organization

PIR Triggers and Timeline#

When PIR is Required#

  • All P0 incidents (within 72 hours)
  • P1 incidents lasting >4 hours (within 1 week)
  • Security incidents (within 48 hours)
  • Customer data impact (within 72 hours)
  • Novel failure modes (any severity, within 1 week)

PIR Timeline#

text
Incident Resolution → 24 hours → PIR Scheduled
Incident Resolution → 72 hours → PIR Data Collection Complete
Incident Resolution → 1 week → PIR Meeting Conducted
PIR Meeting → 1 week → PIR Report Published
PIR Report → 2 weeks → Action Items Assigned
PIR Report → 1 month → Action Items Progress Review

PIR Roles and Responsibilities#

PIR Owner (Incident Commander or delegate)#

  • Schedule PIR meeting within appropriate timeline
  • Collect incident data and prepare timeline
  • Facilitate PIR discussion keeping focus on learnings
  • Write PIR report summarizing findings and actions
  • Track action item completion and effectiveness

PIR Participants#

  • Incident responders (on-call engineer, technical leads, SMEs)
  • Service owners for affected systems
  • Engineering manager for affected teams
  • Product manager (if customer impact)
  • Security representative (for security incidents)
  • Site reliability engineer (observability perspective)

PIR Meeting Facilitator (if different from PIR Owner)#

  • Guide discussion through structured agenda
  • Encourage participation from all attendees
  • Keep focus on systemic issues rather than individual actions
  • Manage time to cover all important areas
  • Capture insights and action items during discussion

PIR Data Collection#

Information Gathering Checklist#

  • Complete incident timeline with timestamps
  • All relevant logs and error messages
  • Monitoring data and graphs covering incident period
  • Chat logs from incident coordination
  • List of all people who participated in response
  • Customer impact data and support ticket volume
  • Business impact assessment (revenue, SLA breaches)
  • External vendor communications if applicable

Timeline Construction#

text
PIR Timeline Template:

[TIME] - Detection
- How was incident first detected?
- What monitoring/alerting fired?
- Who was the first responder?

[TIME] - Investigation Start
- What were the initial symptoms observed?
- What hypotheses were formed?
- What investigation steps were taken?

[TIME] - Key Findings
- When were significant clues discovered?
- How did understanding evolve?
- What false leads were pursued?

[TIME] - Mitigation Attempts
- What fixes were attempted?
- Which worked/didn't work and why?
- Any unintended consequences?

[TIME] - Resolution
- What ultimately resolved the incident?
- How was resolution verified?
- When was normal service confirmed?

[TIME] - Post-Resolution
- Any follow-up issues discovered?
- Customer communication completed?
- Monitoring returned to normal?

Data Preservation#

bash
# Preserve logs for PIR analysis
kubectl logs -l app=[service] --since=[incident-start] > incident-[id]-[service].log

# Export monitoring data
grafana-cli dashboards export [dashboard-id] --time-range=[incident-period]

# Preserve incident communication
slack-export #lilith-incidents --from=[start-time] --to=[end-time]

# Database query logs (if applicable)
kubectl exec postgres-primary-0 -- pg_dump --schema-only > pre-incident-schema.sql

PIR Meeting Structure#

Pre-Meeting Preparation (1 week before)#

  • Schedule 90-minute meeting with all key participants
  • Distribute incident timeline and background materials
  • Ask participants to review materials beforehand
  • Prepare specific questions for each participant
  • Set up collaborative document for note-taking

Meeting Agenda (90 minutes)#

Opening (10 minutes)#

  • PIR philosophy reminder - blameless, learning-focused
  • Incident summary - high-level context for all attendees
  • Meeting objectives - what we want to accomplish
  • Participant introductions - role in incident response

Timeline Review (20 minutes)#

  • Walk through timeline chronologically
  • Validate accuracy with incident responders
  • Fill in gaps where timeline incomplete
  • Clarify decision points and reasoning

Deep Dive Analysis (40 minutes)#

  • Root cause analysis - what caused the incident?
  • Contributing factors - what made it worse or harder to resolve?
  • Response effectiveness - what went well/poorly in response?
  • Communication review - internal and external communication quality

Learnings and Actions (15 minutes)#

  • Key insights - what did we learn?
  • Preventive actions - how to prevent recurrence?
  • Response improvements - how to respond better next time?
  • Monitoring enhancements - how to detect earlier/better?

Wrap-up (5 minutes)#

  • Action item assignment with owners and timelines
  • PIR report timeline commitment
  • Follow-up meeting if needed

PIR Discussion Questions#

Understanding the Incident#

  • What was the sequence of events that led to this incident?
  • Were there any warning signs that we missed?
  • How did the incident progress and evolve?
  • What made this incident particularly challenging?

Response Analysis#

  • How effective was our detection and alerting?
  • Did we escalate appropriately and at the right time?
  • Were we missing any critical information during response?
  • Did we have the right people engaged at the right time?

Contributing Factors#

  • What systemic issues contributed to this incident?
  • Were there process gaps that made response more difficult?
  • Did tool limitations impact our ability to respond?
  • Were there knowledge gaps that slowed resolution?

Learning Opportunities#

  • What surprised us about this incident?
  • What worked particularly well that we should reinforce?
  • What would we do differently if faced with this again?
  • How can we share these learnings with other teams?

PIR Report Structure#

Executive Summary#

text
Incident: [Brief description]
Duration: [Start time] to [resolution time] ([total duration])
Impact: [User impact, business impact, affected services]
Root Cause: [Primary cause with brief explanation]
Resolution: [How incident was resolved]
Key Learnings: [Top 3 insights from incident]
Action Items: [Number of actions with completion timeline]

Detailed Timeline#

text
Complete chronological sequence of events from detection through resolution,
including:
- Decision points and rationale
- Investigation paths taken
- Mitigation attempts
- Communication milestones
- External factors or dependencies

Impact Analysis#

text
User Impact:
- Number of users affected: [specific numbers]
- Duration of impact: [timeline per service]
- Functionality affected: [detailed list]
- Geographic distribution: [regions affected]

Business Impact:
- Revenue impact: [quantified if possible]
- SLA breaches: [which SLAs and duration]
- Customer support volume: [ticket increases]
- Reputation/PR impact: [if applicable]

Technical Impact:
- Services affected: [complete list]
- Data integrity: [any data issues]
- Security implications: [if applicable]
- Performance degradation: [metrics]

Root Cause Analysis#

text
Primary Root Cause:
[Detailed explanation of the fundamental issue that caused the incident]

Contributing Factors:
1. [Factor 1 with explanation]
2. [Factor 2 with explanation]
3. [Factor 3 with explanation]

Why These Factors Existed:
[Analysis of why the contributing factors were present in the system]

Detection and Response Factors:
[What made the incident harder to detect or resolve]

What Went Well#

text
- [Positive aspects of incident response]
- [Effective tools or processes]
- [Good decision-making examples]
- [Effective communication examples]
- [Quick mitigation successes]

What Could Be Improved#

text
- [Areas where response could be better]
- [Process gaps identified]
- [Tool limitations encountered]
- [Communication breakdowns]
- [Knowledge gaps revealed]

Action Items#

Prevention Actions#

text
Action: [Specific action to prevent recurrence]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]

Detection/Response Improvements#

text
Action: [Specific action to improve response]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]

Process/Tool Enhancements#

text
Action: [Specific process or tool improvement]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]

PIR Follow-up Process#

Action Item Tracking#

  • Weekly review of action item progress by PIR owner
  • Bi-weekly updates to stakeholders on progress
  • Monthly review of effectiveness of completed actions
  • Quarterly analysis of trends across multiple PIRs

Action Item Categories#

text
Immediate (1-2 weeks):
- Critical fixes to prevent immediate recurrence
- Hotfixes for dangerous conditions
- Documentation updates for responders

Short-term (1-2 months):
- Process improvements
- Tool enhancements
- Training additions
- Monitoring improvements

Long-term (3-6 months):
- Architecture changes
- System redesigns
- Tool replacements
- Organizational changes

PIR Report Distribution#

text
Required Recipients:
- All incident participants
- Engineering leadership
- Product management (if customer impact)
- Security team (for security incidents)
- Customer support leadership

Optional Recipients:
- Broader engineering teams (for learning)
- Executive team (for P0 incidents)
- Customer success team
- Technical writing team

PIR Quality and Effectiveness#

PIR Success Metrics#

  • Action item completion rate (target: >90% within timeline)
  • Recurrence prevention (same root cause incidents)
  • Response time improvement (faster future incident response)
  • Learning effectiveness (knowledge sharing across teams)

PIR Review Criteria#

  • Timeline is complete and accurate
  • Root cause analysis is thorough and systemic
  • Action items are specific and actionable
  • Learnings are clearly articulated
  • Report is well-written and clear
  • Blameless tone maintained throughout

Common PIR Pitfalls#

Blame-focused Analysis#

  • Focusing on individual mistakes rather than system failures
  • Suggesting individual training as primary solution
  • Using judgmental language about decisions made under pressure

Shallow Analysis#

  • Stopping at proximate cause rather than digging deeper
  • Missing organizational and process contributing factors
  • Not considering why good people made reasonable decisions that led to problems

Weak Action Items#

  • Vague actions that can't be measured
  • No assigned owners or timelines
  • Actions that don't address root causes
  • Too many low-priority actions diluting focus

PIR Improvement Process#

  • Monthly PIR quality review by engineering leadership
  • Quarterly trend analysis across all PIRs
  • Annual PIR process review and updates
  • Training updates based on PIR learnings

PIR Templates and Tools#

PIR Meeting Invite Template#

text
Subject: Post-Incident Review - [Incident Description] - [Date]

We'll be conducting a post-incident review for the [incident description] that occurred on [date].

When: [Date/Time]
Duration: 90 minutes
Where: [Meeting room/video link]

Please review the incident timeline [link] before the meeting.

Agenda:
- Timeline review and validation
- Root cause analysis
- Response effectiveness discussion
- Action item identification

Remember: This is a blameless review focused on learning and improvement.

PIR Report Template#

[Complete template available in separate document: pir-report-template.md]

Action Item Tracking Spreadsheet#

text
Columns: Action ID, Description, Owner, Priority, Due Date, Status, Notes, Completion Date

Remember: The goal of PIR is learning and improvement, not blame. Focus on systemic issues and be honest about what happened to maximize learning for the entire organization.