Purpose: Complete guide for on-call engineers responding to Lilith platform incidents
Pre-Shift Setup#
1. Environment Access Verification#
# Verify Kubernetes access
kubectl get nodes
kubectl get namespaces
# Verify monitoring access
curl -s https://grafana.lilith.com/api/health
# Verify deployment tools
argocd app list
terraform version
2. Communication Setup#
- Join
#lilith-on-callSlack channel - Configure PagerDuty mobile app
- Test incident response phone number
- Verify access to incident management system
3. Tool Access Checklist#
- Grafana dashboards
- ArgoCD deployment console
- Kubernetes cluster access (dev/staging/prod)
- AWS/Cloud provider console
- Vault secrets access
- Database access tools
Alert Response Process#
1. Initial Response (0-5 minutes)#
1. Acknowledge alert in PagerDuty
2. Join #lilith-incidents channel
3. Start incident timer
4. Assess severity using runbook criteria
5. Begin investigation
2. Severity Assessment#
P0 - Critical (Immediate Response)#
- Platform completely down (>50% services failing)
- Data loss or corruption detected
- Security breach confirmed
- Customer data exposure
Response: Immediate acknowledgment, start incident bridge, notify incident commander
P1 - High (30 minutes)#
- Core services degraded (20-50% error rates)
- Significant user impact (>10% of users affected)
- Performance degradation (>5s response times)
- Payment processing issues
Response: Acknowledge within 30 minutes, investigate, update stakeholders
P2 - Medium (2 hours)#
- Non-critical service issues
- Minor performance impact
- Single component failures with redundancy
- Limited user impact (<5% users)
Response: Acknowledge within 2 hours, plan fix during business hours
P3 - Low (Next business day)#
- Monitoring alerts
- Capacity warnings
- Non-user-facing issues
- Maintenance notifications
Response: Acknowledge and triage for next business day
Investigation Procedures#
1. Service Health Check#
# Check all service health endpoints
for service in bff auth content conversation media notification moderation ai; do
echo "Checking $service..."
kubectl get pods -n lilith-$service
curl -f https://$service.lilith.com/v1/health || echo "UNHEALTHY: $service"
done
2. Resource Monitoring#
# Check cluster resources
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory
# Check for failing pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running
3. Log Analysis#
# Check recent errors across services
kubectl logs -l app=bff --tail=100 | grep -i error
kubectl logs -l app=auth --tail=100 | grep -i error
kubectl logs -l app=conversation --tail=100 | grep -i error
# Check specific service logs
kubectl logs deployment/bff-deployment -n lilith-bff --follow
4. Database Health#
# Check database connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT version();"
# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
# Check for long-running queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT query, state, query_start FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;"
Common Issues and Solutions#
BFF Service Issues#
Symptom: High latency or timeouts#
# Check downstream service health
curl -f https://auth.lilith.com/v1/health
curl -f https://conversation.lilith.com/v1/health
curl -f https://ai.lilith.com/v1/health
# Check pod resources
kubectl top pods -n lilith-bff
kubectl describe pods -n lilith-bff
# Check recent deployments
kubectl rollout history deployment/bff-deployment -n lilith-bff
Resolution: Scale up pods or rollback deployment if recent change
Authentication Service Issues#
Symptom: Login failures#
# Check JWT token validation
kubectl logs deployment/auth-deployment -n lilith-auth | grep -i "jwt\|token\|auth"
# Check vault connectivity
kubectl exec -it auth-pod -n lilith-auth -- curl -f https://vault.lilith.com/v1/sys/health
# Verify database connectivity
kubectl exec -it auth-pod -n lilith-auth -- nc -z postgres-service 5432
Resolution: Restart auth pods, verify vault secrets, check database connectivity
AI Service Issues#
Symptom: Model inference failures#
# Check AI service logs
kubectl logs deployment/ai-deployment -n lilith-ai | grep -i "error\|timeout\|model"
# Check GPU resources
kubectl get nodes -o wide
kubectl describe nodes | grep -A5 "nvidia.com/gpu"
# Check model provider connectivity
kubectl exec -it ai-pod -n lilith-ai -- curl -f https://api.openai.com/v1/models
Resolution: Scale GPU nodes, restart model containers, check provider API keys
Database Issues#
Symptom: Connection pool exhaustion#
# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
# Check for blocked queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT blocked_locks.pid AS blocked_pid, blocking_locks.pid AS blocking_pid, blocked_activity.usename AS blocked_user, blocking_activity.usename AS blocking_user FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON(blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid) JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.GRANTED;"
Resolution: Kill long-running queries, scale database resources, adjust connection pool settings
Escalation Procedures#
When to Escalate#
-
Immediate Escalation (P0)
- Unable to resolve within 15 minutes
- Multiple services failing simultaneously
- Security incidents
- Data loss confirmed
-
Timed Escalation (P1)
- No progress after 1 hour
- Customer complaints escalating
- Business-critical features affected
-
Business Hours Escalation (P2/P3)
- Requires architecture changes
- Multiple team coordination needed
- Business decision required
Escalation Contacts#
Platform Lead: @platform-lead (primary), @platform-deputy (secondary)
Security Lead: @security-lead (immediate for security incidents)
Engineering Director: @engineering-director (for P0 after 30 minutes)
Incident Communications#
Internal Updates (Every 30 minutes during P0/P1)#
**Status Update - [Timestamp]**
Incident: [Brief description]
Impact: [User-facing impact]
Progress: [Current investigation/fix status]
ETA: [Estimated resolution time]
Next Update: [Time of next update]
Customer Communications#
P0/P1 Incidents: Update status page within 15 minutes P2 Incidents: Update status page within 2 hours P3 Incidents: No customer communication required
Status Page Template#
We are currently experiencing [issue description].
Our team is actively investigating and working on a resolution.
Affected services: [service list]
We will provide updates every [frequency] until resolved.
Post-Incident Actions#
Immediate (Within 24 hours)#
- Verify full service restoration
- Document timeline in incident log
- Preserve relevant logs and metrics
- Brief next on-call engineer
- Update monitoring if needed
Follow-up (Within 72 hours)#
- Complete post-incident review
- Identify root cause
- Document lessons learned
- Update runbooks if needed
- Schedule follow-up actions
Handoff Procedures#
End of Shift#
- Review active alerts and incidents
- Document any ongoing investigations
- Brief incoming on-call engineer
- Update handoff notes in shared document
- Confirm new engineer has access to all tools
Handoff Checklist Template#
**On-Call Handoff - [Date/Time]**
Active Incidents:
- [List any ongoing incidents]
Ongoing Investigations:
- [List any unresolved alerts or issues]
Recent Changes:
- [Note any recent deployments or changes]
Monitoring Notes:
- [Any trending issues or capacity concerns]
Action Items:
- [Any items requiring follow-up]
Contact Info:
- Outgoing: [your contact]
- Incoming: [incoming contact]
Reference Information#
Key Metrics and Thresholds#
| Metric | Warning | Critical | Action |
|---|---|---|---|
| CPU Usage | >70% | >90% | Scale pods |
| Memory Usage | >80% | >95% | Scale pods |
| Disk Usage | >80% | >90% | Clean up or scale |
| Response Time | >2s | >5s | Investigate/scale |
| Error Rate | >2% | >5% | Immediate investigation |
| Database Connections | >80% of max | >95% of max | Scale/optimize |
Useful Commands Reference#
# Quick service status
kubectl get pods --all-namespaces | grep -v Running
# Resource usage across cluster
kubectl top nodes && kubectl top pods --all-namespaces --sort-by=cpu
# Recent events
kubectl get events --sort-by='.lastTimestamp' --all-namespaces
# Deployment status
kubectl get deployments --all-namespaces
# Service endpoints
kubectl get svc --all-namespaces
# Check pod logs for errors
kubectl logs -l app=<service> --tail=50 | grep -i error
# Scale service
kubectl scale deployment <deployment> --replicas=<count> -n <namespace>
# Rollback deployment
kubectl rollout undo deployment/<deployment> -n <namespace>
Emergency Procedures Summary#
- Service Down: Check health endpoints → Check pods → Check resources → Scale or restart
- High Latency: Check metrics → Check downstream dependencies → Scale if needed
- Database Issues: Check connections → Check queries → Scale or optimize
- Security Alerts: Isolate affected systems → Notify security team → Preserve evidence
- Deployment Issues: Check ArgoCD → Verify manifest → Rollback if needed
Remember: When in doubt, escalate early. It's better to involve additional help than to let an incident grow in severity.