Purpose: Learn from incidents to improve system reliability, response processes, and prevent recurrence
PIR Overview and Philosophy#
Blameless Post-Mortems#
- Focus on systems and processes, not individual actions
- Assume good intentions from all team members
- Encourage transparency to maximize learning
- Multiple contributing factors rather than single root cause
- Systemic improvements over individual accountability
PIR Objectives#
- Understand what happened with complete timeline and context
- Identify contributing factors both technical and organizational
- Extract learnings applicable to future incidents
- Define action items to prevent recurrence and improve response
- Share knowledge across the engineering organization
PIR Triggers and Timeline#
When PIR is Required#
- All P0 incidents (within 72 hours)
- P1 incidents lasting >4 hours (within 1 week)
- Security incidents (within 48 hours)
- Customer data impact (within 72 hours)
- Novel failure modes (any severity, within 1 week)
PIR Timeline#
text
Incident Resolution → 24 hours → PIR Scheduled
Incident Resolution → 72 hours → PIR Data Collection Complete
Incident Resolution → 1 week → PIR Meeting Conducted
PIR Meeting → 1 week → PIR Report Published
PIR Report → 2 weeks → Action Items Assigned
PIR Report → 1 month → Action Items Progress Review
PIR Roles and Responsibilities#
PIR Owner (Incident Commander or delegate)#
- Schedule PIR meeting within appropriate timeline
- Collect incident data and prepare timeline
- Facilitate PIR discussion keeping focus on learnings
- Write PIR report summarizing findings and actions
- Track action item completion and effectiveness
PIR Participants#
- Incident responders (on-call engineer, technical leads, SMEs)
- Service owners for affected systems
- Engineering manager for affected teams
- Product manager (if customer impact)
- Security representative (for security incidents)
- Site reliability engineer (observability perspective)
PIR Meeting Facilitator (if different from PIR Owner)#
- Guide discussion through structured agenda
- Encourage participation from all attendees
- Keep focus on systemic issues rather than individual actions
- Manage time to cover all important areas
- Capture insights and action items during discussion
PIR Data Collection#
Information Gathering Checklist#
- Complete incident timeline with timestamps
- All relevant logs and error messages
- Monitoring data and graphs covering incident period
- Chat logs from incident coordination
- List of all people who participated in response
- Customer impact data and support ticket volume
- Business impact assessment (revenue, SLA breaches)
- External vendor communications if applicable
Timeline Construction#
text
PIR Timeline Template:
[TIME] - Detection
- How was incident first detected?
- What monitoring/alerting fired?
- Who was the first responder?
[TIME] - Investigation Start
- What were the initial symptoms observed?
- What hypotheses were formed?
- What investigation steps were taken?
[TIME] - Key Findings
- When were significant clues discovered?
- How did understanding evolve?
- What false leads were pursued?
[TIME] - Mitigation Attempts
- What fixes were attempted?
- Which worked/didn't work and why?
- Any unintended consequences?
[TIME] - Resolution
- What ultimately resolved the incident?
- How was resolution verified?
- When was normal service confirmed?
[TIME] - Post-Resolution
- Any follow-up issues discovered?
- Customer communication completed?
- Monitoring returned to normal?
Data Preservation#
bash
# Preserve logs for PIR analysis
kubectl logs -l app=[service] --since=[incident-start] > incident-[id]-[service].log
# Export monitoring data
grafana-cli dashboards export [dashboard-id] --time-range=[incident-period]
# Preserve incident communication
slack-export #lilith-incidents --from=[start-time] --to=[end-time]
# Database query logs (if applicable)
kubectl exec postgres-primary-0 -- pg_dump --schema-only > pre-incident-schema.sql
PIR Meeting Structure#
Pre-Meeting Preparation (1 week before)#
- Schedule 90-minute meeting with all key participants
- Distribute incident timeline and background materials
- Ask participants to review materials beforehand
- Prepare specific questions for each participant
- Set up collaborative document for note-taking
Meeting Agenda (90 minutes)#
Opening (10 minutes)#
- PIR philosophy reminder - blameless, learning-focused
- Incident summary - high-level context for all attendees
- Meeting objectives - what we want to accomplish
- Participant introductions - role in incident response
Timeline Review (20 minutes)#
- Walk through timeline chronologically
- Validate accuracy with incident responders
- Fill in gaps where timeline incomplete
- Clarify decision points and reasoning
Deep Dive Analysis (40 minutes)#
- Root cause analysis - what caused the incident?
- Contributing factors - what made it worse or harder to resolve?
- Response effectiveness - what went well/poorly in response?
- Communication review - internal and external communication quality
Learnings and Actions (15 minutes)#
- Key insights - what did we learn?
- Preventive actions - how to prevent recurrence?
- Response improvements - how to respond better next time?
- Monitoring enhancements - how to detect earlier/better?
Wrap-up (5 minutes)#
- Action item assignment with owners and timelines
- PIR report timeline commitment
- Follow-up meeting if needed
PIR Discussion Questions#
Understanding the Incident#
- What was the sequence of events that led to this incident?
- Were there any warning signs that we missed?
- How did the incident progress and evolve?
- What made this incident particularly challenging?
Response Analysis#
- How effective was our detection and alerting?
- Did we escalate appropriately and at the right time?
- Were we missing any critical information during response?
- Did we have the right people engaged at the right time?
Contributing Factors#
- What systemic issues contributed to this incident?
- Were there process gaps that made response more difficult?
- Did tool limitations impact our ability to respond?
- Were there knowledge gaps that slowed resolution?
Learning Opportunities#
- What surprised us about this incident?
- What worked particularly well that we should reinforce?
- What would we do differently if faced with this again?
- How can we share these learnings with other teams?
PIR Report Structure#
Executive Summary#
text
Incident: [Brief description]
Duration: [Start time] to [resolution time] ([total duration])
Impact: [User impact, business impact, affected services]
Root Cause: [Primary cause with brief explanation]
Resolution: [How incident was resolved]
Key Learnings: [Top 3 insights from incident]
Action Items: [Number of actions with completion timeline]
Detailed Timeline#
text
Complete chronological sequence of events from detection through resolution,
including:
- Decision points and rationale
- Investigation paths taken
- Mitigation attempts
- Communication milestones
- External factors or dependencies
Impact Analysis#
text
User Impact:
- Number of users affected: [specific numbers]
- Duration of impact: [timeline per service]
- Functionality affected: [detailed list]
- Geographic distribution: [regions affected]
Business Impact:
- Revenue impact: [quantified if possible]
- SLA breaches: [which SLAs and duration]
- Customer support volume: [ticket increases]
- Reputation/PR impact: [if applicable]
Technical Impact:
- Services affected: [complete list]
- Data integrity: [any data issues]
- Security implications: [if applicable]
- Performance degradation: [metrics]
Root Cause Analysis#
text
Primary Root Cause:
[Detailed explanation of the fundamental issue that caused the incident]
Contributing Factors:
1. [Factor 1 with explanation]
2. [Factor 2 with explanation]
3. [Factor 3 with explanation]
Why These Factors Existed:
[Analysis of why the contributing factors were present in the system]
Detection and Response Factors:
[What made the incident harder to detect or resolve]
What Went Well#
text
- [Positive aspects of incident response]
- [Effective tools or processes]
- [Good decision-making examples]
- [Effective communication examples]
- [Quick mitigation successes]
What Could Be Improved#
text
- [Areas where response could be better]
- [Process gaps identified]
- [Tool limitations encountered]
- [Communication breakdowns]
- [Knowledge gaps revealed]
Action Items#
Prevention Actions#
text
Action: [Specific action to prevent recurrence]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
Detection/Response Improvements#
text
Action: [Specific action to improve response]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
Process/Tool Enhancements#
text
Action: [Specific process or tool improvement]
Owner: [Person/team responsible]
Timeline: [Completion date]
Priority: [High/Medium/Low]
Success Criteria: [How to measure completion]
PIR Follow-up Process#
Action Item Tracking#
- Weekly review of action item progress by PIR owner
- Bi-weekly updates to stakeholders on progress
- Monthly review of effectiveness of completed actions
- Quarterly analysis of trends across multiple PIRs
Action Item Categories#
text
Immediate (1-2 weeks):
- Critical fixes to prevent immediate recurrence
- Hotfixes for dangerous conditions
- Documentation updates for responders
Short-term (1-2 months):
- Process improvements
- Tool enhancements
- Training additions
- Monitoring improvements
Long-term (3-6 months):
- Architecture changes
- System redesigns
- Tool replacements
- Organizational changes
PIR Report Distribution#
text
Required Recipients:
- All incident participants
- Engineering leadership
- Product management (if customer impact)
- Security team (for security incidents)
- Customer support leadership
Optional Recipients:
- Broader engineering teams (for learning)
- Executive team (for P0 incidents)
- Customer success team
- Technical writing team
PIR Quality and Effectiveness#
PIR Success Metrics#
- Action item completion rate (target: >90% within timeline)
- Recurrence prevention (same root cause incidents)
- Response time improvement (faster future incident response)
- Learning effectiveness (knowledge sharing across teams)
PIR Review Criteria#
- Timeline is complete and accurate
- Root cause analysis is thorough and systemic
- Action items are specific and actionable
- Learnings are clearly articulated
- Report is well-written and clear
- Blameless tone maintained throughout
Common PIR Pitfalls#
Blame-focused Analysis#
- Focusing on individual mistakes rather than system failures
- Suggesting individual training as primary solution
- Using judgmental language about decisions made under pressure
Shallow Analysis#
- Stopping at proximate cause rather than digging deeper
- Missing organizational and process contributing factors
- Not considering why good people made reasonable decisions that led to problems
Weak Action Items#
- Vague actions that can't be measured
- No assigned owners or timelines
- Actions that don't address root causes
- Too many low-priority actions diluting focus
PIR Improvement Process#
- Monthly PIR quality review by engineering leadership
- Quarterly trend analysis across all PIRs
- Annual PIR process review and updates
- Training updates based on PIR learnings
PIR Templates and Tools#
PIR Meeting Invite Template#
text
Subject: Post-Incident Review - [Incident Description] - [Date]
We'll be conducting a post-incident review for the [incident description] that occurred on [date].
When: [Date/Time]
Duration: 90 minutes
Where: [Meeting room/video link]
Please review the incident timeline [link] before the meeting.
Agenda:
- Timeline review and validation
- Root cause analysis
- Response effectiveness discussion
- Action item identification
Remember: This is a blameless review focused on learning and improvement.
PIR Report Template#
[Complete template available in separate document: pir-report-template.md]
Action Item Tracking Spreadsheet#
text
Columns: Action ID, Description, Owner, Priority, Due Date, Status, Notes, Completion Date
Remember: The goal of PIR is learning and improvement, not blame. Focus on systemic issues and be honest about what happened to maximize learning for the entire organization.