# On-Call Handoff Checklist

**Purpose**: Structured handoff process between on-call engineers to ensure
continuity and no missed incidents

## Pre-Handoff Preparation (30 minutes before shift end)

### 1. Review Active Incidents

- [ ] Check PagerDuty for any active incidents
- [ ] Review #lilith-incidents channel for ongoing issues
- [ ] Document current status of any active investigations
- [ ] Identify any incidents requiring immediate attention

### 2. System Health Review

- [ ] Check all service health dashboards in Grafana
- [ ] Review recent alerts and their resolution status
- [ ] Verify all monitoring systems are operational
- [ ] Check for any trending issues or capacity concerns

### 3. Recent Changes Assessment

- [ ] Review recent deployments in ArgoCD
- [ ] Check #lilith-deployments channel for recent changes
- [ ] Identify any deployments that might need monitoring
- [ ] Note any configuration changes or maintenance windows

## Handoff Meeting (15 minutes)

### Attendees

- Outgoing on-call engineer
- Incoming on-call engineer
- Platform lead (for critical handoffs only)

### Agenda Template

```
**On-Call Handoff Meeting**
Date: [Date]
Time: [Time]
Outgoing: [Name]
Incoming: [Name]

1. Active Incidents Review (5 minutes)
2. System Status Update (5 minutes)
3. Action Items and Follow-ups (3 minutes)
4. Questions and Clarifications (2 minutes)
```

## Handoff Documentation

### Handoff Form Template

```markdown
# On-Call Handoff Report

**Handoff Details**

- Date/Time: [YYYY-MM-DD HH:MM UTC]
- Outgoing Engineer: [Name] - [Contact Info]
- Incoming Engineer: [Name] - [Contact Info]
- Duration of Shift: [Start] to [End]

## Active Incidents

### Incident #1 (if any)

- **Status**: [Active/Investigating/Monitoring]
- **Severity**: [P0/P1/P2/P3]
- **Description**: [Brief incident description]
- **Impact**: [User/service impact]
- **Current Actions**: [What's being done]
- **Next Steps**: [Required follow-up actions]
- **Timeline**: [Key events and timeline]

### Incident #2 (if any)

[Same format as above]

## System Health Summary

### Service Status

- **BFF**: [Healthy/Degraded/Down] - [Notes if any]
- **Auth**: [Healthy/Degraded/Down] - [Notes if any]
- **Content**: [Healthy/Degraded/Down] - [Notes if any]
- **Conversation**: [Healthy/Degraded/Down] - [Notes if any]
- **Media**: [Healthy/Degraded/Down] - [Notes if any]
- **AI**: [Healthy/Degraded/Down] - [Notes if any]
- **Notification**: [Healthy/Degraded/Down] - [Notes if any]
- **Moderation**: [Healthy/Degraded/Down] - [Notes if any]

### Infrastructure Status

- **Kubernetes Clusters**: [Status and notes]
- **Database**: [Status and performance notes]
- **Monitoring**: [Grafana/Prometheus status]
- **External Dependencies**: [Third-party service status]

### Key Metrics Snapshot

- **Overall Error Rate**: [Percentage]
- **Average Response Time**: [Milliseconds]
- **Database Connection Pool**: [Usage percentage]
- **CPU/Memory Utilization**: [Cluster averages]

## Recent Changes and Deployments

### Deployments (Last 24 hours)

- **Service**: [Service name] - **Version**: [Version] - **Time**: [Deployment
  time] - **Status**: [Success/Failed/Monitoring]
- [Additional deployments]

### Configuration Changes

- [List any configuration changes made during shift]

### Infrastructure Changes

- [List any infrastructure modifications]

## Resolved Issues During Shift

### Issue #1

- **Time**: [When resolved]
- **Description**: [What was the issue]
- **Resolution**: [How it was fixed]
- **Root Cause**: [If identified]

### Issue #2

[Same format]

## Ongoing Concerns and Monitoring

### Items Requiring Continued Monitoring

- [List items that need watching]
- [Include expected timeframes and thresholds]

### Trending Issues

- [Note any patterns or developing issues]
- [Include recommended preventive actions]

### Capacity Concerns

- [Note any resource constraints]
- [Include scaling recommendations]

## Action Items for Incoming Engineer

### Immediate (Next 4 hours)

- [ ] [Action item 1 with deadline]
- [ ] [Action item 2 with deadline]

### During Shift (Next 24 hours)

- [ ] [Action item 1]
- [ ] [Action item 2]

### Follow-up Required

- [ ] [Item requiring business hours follow-up]
- [ ] [Item requiring team coordination]

## Notes and Observations

### Lessons Learned

- [Any insights gained during the shift]

### Process Improvements

- [Suggestions for improving procedures]

### Tool/System Issues

- [Any problems with monitoring or response tools]

## Contact Information

### Outgoing Engineer

- **Primary**: [Phone number]
- **Secondary**: [Alternative contact]
- **Available Until**: [Time when no longer available]

### Key Contacts (if escalation needed)

- **Platform Lead**: [Contact info]
- **Security Lead**: [Contact info]
- **Engineering Director**: [Contact info]

## Handoff Confirmation

- [ ] Incoming engineer has reviewed all active incidents
- [ ] Incoming engineer has access to all required tools
- [ ] Incoming engineer understands priority action items
- [ ] Contact information has been exchanged
- [ ] Handoff documented and shared with team

**Signatures** Outgoing Engineer: [Name] - [Timestamp] Incoming Engineer:
[Name] - [Timestamp]
```

## Post-Handoff Actions

### Immediate (Within 1 hour)

- [ ] Save handoff document to shared drive
- [ ] Post handoff summary in #lilith-on-call channel
- [ ] Update on-call calendar with any shift changes
- [ ] Confirm incoming engineer can access all systems

### Follow-up (Within 24 hours)

- [ ] Review handoff for any missed items
- [ ] Update runbooks based on shift learnings
- [ ] Complete any incident documentation
- [ ] Schedule post-incident reviews if needed

## Handoff Quality Checklist

### Information Completeness

- [ ] All active incidents documented with sufficient detail
- [ ] Current system status clearly communicated
- [ ] All action items prioritized and explained
- [ ] Contact information provided and verified

### Knowledge Transfer

- [ ] Incoming engineer understands context of ongoing issues
- [ ] Investigation steps and findings clearly explained
- [ ] Escalation triggers and contacts clarified
- [ ] Questions answered satisfactorily

### Tool Access Verification

- [ ] Incoming engineer can access PagerDuty
- [ ] Grafana dashboards accessible
- [ ] Kubernetes cluster access confirmed
- [ ] ArgoCD and deployment tools working
- [ ] Communication channels joined

## Emergency Handoff Procedures

### If Outgoing Engineer Becomes Unavailable

1. Platform lead assumes responsibility
2. Check most recent handoff documentation
3. Review recent PagerDuty and Slack activity
4. Contact backup on-call engineer
5. Escalate to engineering director if needed

### If Incoming Engineer Cannot Take Shift

1. Outgoing engineer continues until replacement found
2. Notify platform lead immediately
3. Activate backup on-call rotation
4. Document extended shift in handoff notes
5. Ensure adequate rest before next scheduled shift

## Handoff Best Practices

### Do's

- Be thorough but concise in documentation
- Ask questions if anything is unclear
- Test critical system access during handoff
- Provide context, not just facts
- Share intuition about system behavior

### Don'ts

- Rush through the handoff process
- Assume knowledge without verification
- Leave action items without clear priorities
- Skip documentation even for "quiet" shifts
- Hand off active incidents without proper context

## Quality Metrics

### Handoff Effectiveness Indicators

- Time to resolve first incident (incoming engineer)
- Number of escalations due to missed information
- Feedback scores from incoming engineers
- Incident response time improvements

### Continuous Improvement

- Monthly review of handoff quality
- Feedback collection from on-call engineers
- Process refinement based on lessons learned
- Tool improvements to support better handoffs

---

**Note**: A good handoff can prevent incidents and reduce resolution time.
Invest the time to do it properly.
