Eve library

On-Call Engineer Guide

Runbook

Revision 1

On this page

Purpose: Complete guide for on-call engineers responding to Lilith platform incidents

Pre-Shift Setup#

1. Environment Access Verification#

bash
# Verify Kubernetes access
kubectl get nodes
kubectl get namespaces

# Verify monitoring access
curl -s https://grafana.lilith.com/api/health

# Verify deployment tools
argocd app list
terraform version

2. Communication Setup#

  • Join #lilith-on-call Slack channel
  • Configure PagerDuty mobile app
  • Test incident response phone number
  • Verify access to incident management system

3. Tool Access Checklist#

  • Grafana dashboards
  • ArgoCD deployment console
  • Kubernetes cluster access (dev/staging/prod)
  • AWS/Cloud provider console
  • Vault secrets access
  • Database access tools

Alert Response Process#

1. Initial Response (0-5 minutes)#

text
1. Acknowledge alert in PagerDuty
2. Join #lilith-incidents channel
3. Start incident timer
4. Assess severity using runbook criteria
5. Begin investigation

2. Severity Assessment#

P0 - Critical (Immediate Response)#

  • Platform completely down (>50% services failing)
  • Data loss or corruption detected
  • Security breach confirmed
  • Customer data exposure

Response: Immediate acknowledgment, start incident bridge, notify incident commander

P1 - High (30 minutes)#

  • Core services degraded (20-50% error rates)
  • Significant user impact (>10% of users affected)
  • Performance degradation (>5s response times)
  • Payment processing issues

Response: Acknowledge within 30 minutes, investigate, update stakeholders

P2 - Medium (2 hours)#

  • Non-critical service issues
  • Minor performance impact
  • Single component failures with redundancy
  • Limited user impact (<5% users)

Response: Acknowledge within 2 hours, plan fix during business hours

P3 - Low (Next business day)#

  • Monitoring alerts
  • Capacity warnings
  • Non-user-facing issues
  • Maintenance notifications

Response: Acknowledge and triage for next business day

Investigation Procedures#

1. Service Health Check#

bash
# Check all service health endpoints
for service in bff auth content conversation media notification moderation ai; do
  echo "Checking $service..."
  kubectl get pods -n lilith-$service
  curl -f https://$service.lilith.com/v1/health || echo "UNHEALTHY: $service"
done

2. Resource Monitoring#

bash
# Check cluster resources
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory

# Check for failing pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

3. Log Analysis#

bash
# Check recent errors across services
kubectl logs -l app=bff --tail=100 | grep -i error
kubectl logs -l app=auth --tail=100 | grep -i error
kubectl logs -l app=conversation --tail=100 | grep -i error

# Check specific service logs
kubectl logs deployment/bff-deployment -n lilith-bff --follow

4. Database Health#

bash
# Check database connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT version();"

# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

# Check for long-running queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT query, state, query_start FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;"

Common Issues and Solutions#

BFF Service Issues#

Symptom: High latency or timeouts#

bash
# Check downstream service health
curl -f https://auth.lilith.com/v1/health
curl -f https://conversation.lilith.com/v1/health
curl -f https://ai.lilith.com/v1/health

# Check pod resources
kubectl top pods -n lilith-bff
kubectl describe pods -n lilith-bff

# Check recent deployments
kubectl rollout history deployment/bff-deployment -n lilith-bff

Resolution: Scale up pods or rollback deployment if recent change

Authentication Service Issues#

Symptom: Login failures#

bash
# Check JWT token validation
kubectl logs deployment/auth-deployment -n lilith-auth | grep -i "jwt\|token\|auth"

# Check vault connectivity
kubectl exec -it auth-pod -n lilith-auth -- curl -f https://vault.lilith.com/v1/sys/health

# Verify database connectivity
kubectl exec -it auth-pod -n lilith-auth -- nc -z postgres-service 5432

Resolution: Restart auth pods, verify vault secrets, check database connectivity

AI Service Issues#

Symptom: Model inference failures#

bash
# Check AI service logs
kubectl logs deployment/ai-deployment -n lilith-ai | grep -i "error\|timeout\|model"

# Check GPU resources
kubectl get nodes -o wide
kubectl describe nodes | grep -A5 "nvidia.com/gpu"

# Check model provider connectivity
kubectl exec -it ai-pod -n lilith-ai -- curl -f https://api.openai.com/v1/models

Resolution: Scale GPU nodes, restart model containers, check provider API keys

Database Issues#

Symptom: Connection pool exhaustion#

bash
# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

# Check for blocked queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT blocked_locks.pid AS blocked_pid, blocking_locks.pid AS blocking_pid, blocked_activity.usename AS blocked_user, blocking_activity.usename AS blocking_user FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON(blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid) JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.GRANTED;"

Resolution: Kill long-running queries, scale database resources, adjust connection pool settings

Escalation Procedures#

When to Escalate#

  1. Immediate Escalation (P0)

    • Unable to resolve within 15 minutes
    • Multiple services failing simultaneously
    • Security incidents
    • Data loss confirmed
  2. Timed Escalation (P1)

    • No progress after 1 hour
    • Customer complaints escalating
    • Business-critical features affected
  3. Business Hours Escalation (P2/P3)

    • Requires architecture changes
    • Multiple team coordination needed
    • Business decision required

Escalation Contacts#

text
Platform Lead: @platform-lead (primary), @platform-deputy (secondary)
Security Lead: @security-lead (immediate for security incidents)
Engineering Director: @engineering-director (for P0 after 30 minutes)

Incident Communications#

Internal Updates (Every 30 minutes during P0/P1)#

text
**Status Update - [Timestamp]**
Incident: [Brief description]
Impact: [User-facing impact]
Progress: [Current investigation/fix status]
ETA: [Estimated resolution time]
Next Update: [Time of next update]

Customer Communications#

P0/P1 Incidents: Update status page within 15 minutes P2 Incidents: Update status page within 2 hours P3 Incidents: No customer communication required

Status Page Template#

text
We are currently experiencing [issue description].
Our team is actively investigating and working on a resolution.
Affected services: [service list]
We will provide updates every [frequency] until resolved.

Post-Incident Actions#

Immediate (Within 24 hours)#

  • Verify full service restoration
  • Document timeline in incident log
  • Preserve relevant logs and metrics
  • Brief next on-call engineer
  • Update monitoring if needed

Follow-up (Within 72 hours)#

  • Complete post-incident review
  • Identify root cause
  • Document lessons learned
  • Update runbooks if needed
  • Schedule follow-up actions

Handoff Procedures#

End of Shift#

  1. Review active alerts and incidents
  2. Document any ongoing investigations
  3. Brief incoming on-call engineer
  4. Update handoff notes in shared document
  5. Confirm new engineer has access to all tools

Handoff Checklist Template#

text
**On-Call Handoff - [Date/Time]**

Active Incidents:
- [List any ongoing incidents]

Ongoing Investigations:
- [List any unresolved alerts or issues]

Recent Changes:
- [Note any recent deployments or changes]

Monitoring Notes:
- [Any trending issues or capacity concerns]

Action Items:
- [Any items requiring follow-up]

Contact Info:
- Outgoing: [your contact]
- Incoming: [incoming contact]

Reference Information#

Key Metrics and Thresholds#

Metric Warning Critical Action
CPU Usage >70% >90% Scale pods
Memory Usage >80% >95% Scale pods
Disk Usage >80% >90% Clean up or scale
Response Time >2s >5s Investigate/scale
Error Rate >2% >5% Immediate investigation
Database Connections >80% of max >95% of max Scale/optimize

Useful Commands Reference#

bash
# Quick service status
kubectl get pods --all-namespaces | grep -v Running

# Resource usage across cluster
kubectl top nodes && kubectl top pods --all-namespaces --sort-by=cpu

# Recent events
kubectl get events --sort-by='.lastTimestamp' --all-namespaces

# Deployment status
kubectl get deployments --all-namespaces

# Service endpoints
kubectl get svc --all-namespaces

# Check pod logs for errors
kubectl logs -l app=<service> --tail=50 | grep -i error

# Scale service
kubectl scale deployment <deployment> --replicas=<count> -n <namespace>

# Rollback deployment
kubectl rollout undo deployment/<deployment> -n <namespace>

Emergency Procedures Summary#

  1. Service Down: Check health endpoints → Check pods → Check resources → Scale or restart
  2. High Latency: Check metrics → Check downstream dependencies → Scale if needed
  3. Database Issues: Check connections → Check queries → Scale or optimize
  4. Security Alerts: Isolate affected systems → Notify security team → Preserve evidence
  5. Deployment Issues: Check ArgoCD → Verify manifest → Rollback if needed

Remember: When in doubt, escalate early. It's better to involve additional help than to let an incident grow in severity.