Purpose: Structured handoff process between on-call engineers to ensure continuity and no missed incidents
Pre-Handoff Preparation (30 minutes before shift end)#
1. Review Active Incidents#
- Check PagerDuty for any active incidents
- Review #lilith-incidents channel for ongoing issues
- Document current status of any active investigations
- Identify any incidents requiring immediate attention
2. System Health Review#
- Check all service health dashboards in Grafana
- Review recent alerts and their resolution status
- Verify all monitoring systems are operational
- Check for any trending issues or capacity concerns
3. Recent Changes Assessment#
- Review recent deployments in ArgoCD
- Check #lilith-deployments channel for recent changes
- Identify any deployments that might need monitoring
- Note any configuration changes or maintenance windows
Handoff Meeting (15 minutes)#
Attendees#
- Outgoing on-call engineer
- Incoming on-call engineer
- Platform lead (for critical handoffs only)
Agenda Template#
text
**On-Call Handoff Meeting**
Date: [Date]
Time: [Time]
Outgoing: [Name]
Incoming: [Name]
1. Active Incidents Review (5 minutes)
2. System Status Update (5 minutes)
3. Action Items and Follow-ups (3 minutes)
4. Questions and Clarifications (2 minutes)
Handoff Documentation#
Handoff Form Template#
markdown
# On-Call Handoff Report
**Handoff Details**
- Date/Time: [YYYY-MM-DD HH:MM UTC]
- Outgoing Engineer: [Name] - [Contact Info]
- Incoming Engineer: [Name] - [Contact Info]
- Duration of Shift: [Start] to [End]
## Active Incidents
### Incident #1 (if any)
- **Status**: [Active/Investigating/Monitoring]
- **Severity**: [P0/P1/P2/P3]
- **Description**: [Brief incident description]
- **Impact**: [User/service impact]
- **Current Actions**: [What's being done]
- **Next Steps**: [Required follow-up actions]
- **Timeline**: [Key events and timeline]
### Incident #2 (if any)
[Same format as above]
## System Health Summary
### Service Status
- **BFF**: [Healthy/Degraded/Down] - [Notes if any]
- **Auth**: [Healthy/Degraded/Down] - [Notes if any]
- **Content**: [Healthy/Degraded/Down] - [Notes if any]
- **Conversation**: [Healthy/Degraded/Down] - [Notes if any]
- **Media**: [Healthy/Degraded/Down] - [Notes if any]
- **AI**: [Healthy/Degraded/Down] - [Notes if any]
- **Notification**: [Healthy/Degraded/Down] - [Notes if any]
- **Moderation**: [Healthy/Degraded/Down] - [Notes if any]
### Infrastructure Status
- **Kubernetes Clusters**: [Status and notes]
- **Database**: [Status and performance notes]
- **Monitoring**: [Grafana/Prometheus status]
- **External Dependencies**: [Third-party service status]
### Key Metrics Snapshot
- **Overall Error Rate**: [Percentage]
- **Average Response Time**: [Milliseconds]
- **Database Connection Pool**: [Usage percentage]
- **CPU/Memory Utilization**: [Cluster averages]
## Recent Changes and Deployments
### Deployments (Last 24 hours)
- **Service**: [Service name] - **Version**: [Version] - **Time**: [Deployment
time] - **Status**: [Success/Failed/Monitoring]
- [Additional deployments]
### Configuration Changes
- [List any configuration changes made during shift]
### Infrastructure Changes
- [List any infrastructure modifications]
## Resolved Issues During Shift
### Issue #1
- **Time**: [When resolved]
- **Description**: [What was the issue]
- **Resolution**: [How it was fixed]
- **Root Cause**: [If identified]
### Issue #2
[Same format]
## Ongoing Concerns and Monitoring
### Items Requiring Continued Monitoring
- [List items that need watching]
- [Include expected timeframes and thresholds]
### Trending Issues
- [Note any patterns or developing issues]
- [Include recommended preventive actions]
### Capacity Concerns
- [Note any resource constraints]
- [Include scaling recommendations]
## Action Items for Incoming Engineer
### Immediate (Next 4 hours)
- [ ] [Action item 1 with deadline]
- [ ] [Action item 2 with deadline]
### During Shift (Next 24 hours)
- [ ] [Action item 1]
- [ ] [Action item 2]
### Follow-up Required
- [ ] [Item requiring business hours follow-up]
- [ ] [Item requiring team coordination]
## Notes and Observations
### Lessons Learned
- [Any insights gained during the shift]
### Process Improvements
- [Suggestions for improving procedures]
### Tool/System Issues
- [Any problems with monitoring or response tools]
## Contact Information
### Outgoing Engineer
- **Primary**: [Phone number]
- **Secondary**: [Alternative contact]
- **Available Until**: [Time when no longer available]
### Key Contacts (if escalation needed)
- **Platform Lead**: [Contact info]
- **Security Lead**: [Contact info]
- **Engineering Director**: [Contact info]
## Handoff Confirmation
- [ ] Incoming engineer has reviewed all active incidents
- [ ] Incoming engineer has access to all required tools
- [ ] Incoming engineer understands priority action items
- [ ] Contact information has been exchanged
- [ ] Handoff documented and shared with team
**Signatures** Outgoing Engineer: [Name] - [Timestamp] Incoming Engineer:
[Name] - [Timestamp]
Post-Handoff Actions#
Immediate (Within 1 hour)#
- Save handoff document to shared drive
- Post handoff summary in #lilith-on-call channel
- Update on-call calendar with any shift changes
- Confirm incoming engineer can access all systems
Follow-up (Within 24 hours)#
- Review handoff for any missed items
- Update runbooks based on shift learnings
- Complete any incident documentation
- Schedule post-incident reviews if needed
Handoff Quality Checklist#
Information Completeness#
- All active incidents documented with sufficient detail
- Current system status clearly communicated
- All action items prioritized and explained
- Contact information provided and verified
Knowledge Transfer#
- Incoming engineer understands context of ongoing issues
- Investigation steps and findings clearly explained
- Escalation triggers and contacts clarified
- Questions answered satisfactorily
Tool Access Verification#
- Incoming engineer can access PagerDuty
- Grafana dashboards accessible
- Kubernetes cluster access confirmed
- ArgoCD and deployment tools working
- Communication channels joined
Emergency Handoff Procedures#
If Outgoing Engineer Becomes Unavailable#
- Platform lead assumes responsibility
- Check most recent handoff documentation
- Review recent PagerDuty and Slack activity
- Contact backup on-call engineer
- Escalate to engineering director if needed
If Incoming Engineer Cannot Take Shift#
- Outgoing engineer continues until replacement found
- Notify platform lead immediately
- Activate backup on-call rotation
- Document extended shift in handoff notes
- Ensure adequate rest before next scheduled shift
Handoff Best Practices#
Do's#
- Be thorough but concise in documentation
- Ask questions if anything is unclear
- Test critical system access during handoff
- Provide context, not just facts
- Share intuition about system behavior
Don'ts#
- Rush through the handoff process
- Assume knowledge without verification
- Leave action items without clear priorities
- Skip documentation even for "quiet" shifts
- Hand off active incidents without proper context
Quality Metrics#
Handoff Effectiveness Indicators#
- Time to resolve first incident (incoming engineer)
- Number of escalations due to missed information
- Feedback scores from incoming engineers
- Incident response time improvements
Continuous Improvement#
- Monthly review of handoff quality
- Feedback collection from on-call engineers
- Process refinement based on lessons learned
- Tool improvements to support better handoffs
Note: A good handoff can prevent incidents and reduce resolution time. Invest the time to do it properly.