Eve library

On-Call Handoff Checklist

Runbook

Revision 1

On this page

Purpose: Structured handoff process between on-call engineers to ensure continuity and no missed incidents

Pre-Handoff Preparation (30 minutes before shift end)#

1. Review Active Incidents#

  • Check PagerDuty for any active incidents
  • Review #lilith-incidents channel for ongoing issues
  • Document current status of any active investigations
  • Identify any incidents requiring immediate attention

2. System Health Review#

  • Check all service health dashboards in Grafana
  • Review recent alerts and their resolution status
  • Verify all monitoring systems are operational
  • Check for any trending issues or capacity concerns

3. Recent Changes Assessment#

  • Review recent deployments in ArgoCD
  • Check #lilith-deployments channel for recent changes
  • Identify any deployments that might need monitoring
  • Note any configuration changes or maintenance windows

Handoff Meeting (15 minutes)#

Attendees#

  • Outgoing on-call engineer
  • Incoming on-call engineer
  • Platform lead (for critical handoffs only)

Agenda Template#

text
**On-Call Handoff Meeting**
Date: [Date]
Time: [Time]
Outgoing: [Name]
Incoming: [Name]

1. Active Incidents Review (5 minutes)
2. System Status Update (5 minutes)
3. Action Items and Follow-ups (3 minutes)
4. Questions and Clarifications (2 minutes)

Handoff Documentation#

Handoff Form Template#

markdown
# On-Call Handoff Report

**Handoff Details**

- Date/Time: [YYYY-MM-DD HH:MM UTC]
- Outgoing Engineer: [Name] - [Contact Info]
- Incoming Engineer: [Name] - [Contact Info]
- Duration of Shift: [Start] to [End]

## Active Incidents

### Incident #1 (if any)

- **Status**: [Active/Investigating/Monitoring]
- **Severity**: [P0/P1/P2/P3]
- **Description**: [Brief incident description]
- **Impact**: [User/service impact]
- **Current Actions**: [What's being done]
- **Next Steps**: [Required follow-up actions]
- **Timeline**: [Key events and timeline]

### Incident #2 (if any)

[Same format as above]

## System Health Summary

### Service Status

- **BFF**: [Healthy/Degraded/Down] - [Notes if any]
- **Auth**: [Healthy/Degraded/Down] - [Notes if any]
- **Content**: [Healthy/Degraded/Down] - [Notes if any]
- **Conversation**: [Healthy/Degraded/Down] - [Notes if any]
- **Media**: [Healthy/Degraded/Down] - [Notes if any]
- **AI**: [Healthy/Degraded/Down] - [Notes if any]
- **Notification**: [Healthy/Degraded/Down] - [Notes if any]
- **Moderation**: [Healthy/Degraded/Down] - [Notes if any]

### Infrastructure Status

- **Kubernetes Clusters**: [Status and notes]
- **Database**: [Status and performance notes]
- **Monitoring**: [Grafana/Prometheus status]
- **External Dependencies**: [Third-party service status]

### Key Metrics Snapshot

- **Overall Error Rate**: [Percentage]
- **Average Response Time**: [Milliseconds]
- **Database Connection Pool**: [Usage percentage]
- **CPU/Memory Utilization**: [Cluster averages]

## Recent Changes and Deployments

### Deployments (Last 24 hours)

- **Service**: [Service name] - **Version**: [Version] - **Time**: [Deployment
  time] - **Status**: [Success/Failed/Monitoring]
- [Additional deployments]

### Configuration Changes

- [List any configuration changes made during shift]

### Infrastructure Changes

- [List any infrastructure modifications]

## Resolved Issues During Shift

### Issue #1

- **Time**: [When resolved]
- **Description**: [What was the issue]
- **Resolution**: [How it was fixed]
- **Root Cause**: [If identified]

### Issue #2

[Same format]

## Ongoing Concerns and Monitoring

### Items Requiring Continued Monitoring

- [List items that need watching]
- [Include expected timeframes and thresholds]

### Trending Issues

- [Note any patterns or developing issues]
- [Include recommended preventive actions]

### Capacity Concerns

- [Note any resource constraints]
- [Include scaling recommendations]

## Action Items for Incoming Engineer

### Immediate (Next 4 hours)

- [ ] [Action item 1 with deadline]
- [ ] [Action item 2 with deadline]

### During Shift (Next 24 hours)

- [ ] [Action item 1]
- [ ] [Action item 2]

### Follow-up Required

- [ ] [Item requiring business hours follow-up]
- [ ] [Item requiring team coordination]

## Notes and Observations

### Lessons Learned

- [Any insights gained during the shift]

### Process Improvements

- [Suggestions for improving procedures]

### Tool/System Issues

- [Any problems with monitoring or response tools]

## Contact Information

### Outgoing Engineer

- **Primary**: [Phone number]
- **Secondary**: [Alternative contact]
- **Available Until**: [Time when no longer available]

### Key Contacts (if escalation needed)

- **Platform Lead**: [Contact info]
- **Security Lead**: [Contact info]
- **Engineering Director**: [Contact info]

## Handoff Confirmation

- [ ] Incoming engineer has reviewed all active incidents
- [ ] Incoming engineer has access to all required tools
- [ ] Incoming engineer understands priority action items
- [ ] Contact information has been exchanged
- [ ] Handoff documented and shared with team

**Signatures** Outgoing Engineer: [Name] - [Timestamp] Incoming Engineer:
[Name] - [Timestamp]

Post-Handoff Actions#

Immediate (Within 1 hour)#

  • Save handoff document to shared drive
  • Post handoff summary in #lilith-on-call channel
  • Update on-call calendar with any shift changes
  • Confirm incoming engineer can access all systems

Follow-up (Within 24 hours)#

  • Review handoff for any missed items
  • Update runbooks based on shift learnings
  • Complete any incident documentation
  • Schedule post-incident reviews if needed

Handoff Quality Checklist#

Information Completeness#

  • All active incidents documented with sufficient detail
  • Current system status clearly communicated
  • All action items prioritized and explained
  • Contact information provided and verified

Knowledge Transfer#

  • Incoming engineer understands context of ongoing issues
  • Investigation steps and findings clearly explained
  • Escalation triggers and contacts clarified
  • Questions answered satisfactorily

Tool Access Verification#

  • Incoming engineer can access PagerDuty
  • Grafana dashboards accessible
  • Kubernetes cluster access confirmed
  • ArgoCD and deployment tools working
  • Communication channels joined

Emergency Handoff Procedures#

If Outgoing Engineer Becomes Unavailable#

  1. Platform lead assumes responsibility
  2. Check most recent handoff documentation
  3. Review recent PagerDuty and Slack activity
  4. Contact backup on-call engineer
  5. Escalate to engineering director if needed

If Incoming Engineer Cannot Take Shift#

  1. Outgoing engineer continues until replacement found
  2. Notify platform lead immediately
  3. Activate backup on-call rotation
  4. Document extended shift in handoff notes
  5. Ensure adequate rest before next scheduled shift

Handoff Best Practices#

Do's#

  • Be thorough but concise in documentation
  • Ask questions if anything is unclear
  • Test critical system access during handoff
  • Provide context, not just facts
  • Share intuition about system behavior

Don'ts#

  • Rush through the handoff process
  • Assume knowledge without verification
  • Leave action items without clear priorities
  • Skip documentation even for "quiet" shifts
  • Hand off active incidents without proper context

Quality Metrics#

Handoff Effectiveness Indicators#

  • Time to resolve first incident (incoming engineer)
  • Number of escalations due to missed information
  • Feedback scores from incoming engineers
  • Incident response time improvements

Continuous Improvement#

  • Monthly review of handoff quality
  • Feedback collection from on-call engineers
  • Process refinement based on lessons learned
  • Tool improvements to support better handoffs

Note: A good handoff can prevent incidents and reduce resolution time. Invest the time to do it properly.