# On-Call Engineer Guide

**Purpose**: Complete guide for on-call engineers responding to Lilith platform
incidents

## Pre-Shift Setup

### 1. Environment Access Verification

```bash
# Verify Kubernetes access
kubectl get nodes
kubectl get namespaces

# Verify monitoring access
curl -s https://grafana.lilith.com/api/health

# Verify deployment tools
argocd app list
terraform version
```

### 2. Communication Setup

- [ ] Join `#lilith-on-call` Slack channel
- [ ] Configure PagerDuty mobile app
- [ ] Test incident response phone number
- [ ] Verify access to incident management system

### 3. Tool Access Checklist

- [ ] Grafana dashboards
- [ ] ArgoCD deployment console
- [ ] Kubernetes cluster access (dev/staging/prod)
- [ ] AWS/Cloud provider console
- [ ] Vault secrets access
- [ ] Database access tools

## Alert Response Process

### 1. Initial Response (0-5 minutes)

```
1. Acknowledge alert in PagerDuty
2. Join #lilith-incidents channel
3. Start incident timer
4. Assess severity using runbook criteria
5. Begin investigation
```

### 2. Severity Assessment

#### P0 - Critical (Immediate Response)

- Platform completely down (>50% services failing)
- Data loss or corruption detected
- Security breach confirmed
- Customer data exposure

**Response**: Immediate acknowledgment, start incident bridge, notify incident
commander

#### P1 - High (30 minutes)

- Core services degraded (20-50% error rates)
- Significant user impact (>10% of users affected)
- Performance degradation (>5s response times)
- Payment processing issues

**Response**: Acknowledge within 30 minutes, investigate, update stakeholders

#### P2 - Medium (2 hours)

- Non-critical service issues
- Minor performance impact
- Single component failures with redundancy
- Limited user impact (<5% users)

**Response**: Acknowledge within 2 hours, plan fix during business hours

#### P3 - Low (Next business day)

- Monitoring alerts
- Capacity warnings
- Non-user-facing issues
- Maintenance notifications

**Response**: Acknowledge and triage for next business day

## Investigation Procedures

### 1. Service Health Check

```bash
# Check all service health endpoints
for service in bff auth content conversation media notification moderation ai; do
  echo "Checking $service..."
  kubectl get pods -n lilith-$service
  curl -f https://$service.lilith.com/v1/health || echo "UNHEALTHY: $service"
done
```

### 2. Resource Monitoring

```bash
# Check cluster resources
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory

# Check for failing pods
kubectl get pods --all-namespaces --field-selector=status.phase!=Running
```

### 3. Log Analysis

```bash
# Check recent errors across services
kubectl logs -l app=bff --tail=100 | grep -i error
kubectl logs -l app=auth --tail=100 | grep -i error
kubectl logs -l app=conversation --tail=100 | grep -i error

# Check specific service logs
kubectl logs deployment/bff-deployment -n lilith-bff --follow
```

### 4. Database Health

```bash
# Check database connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT version();"

# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

# Check for long-running queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT query, state, query_start FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;"
```

## Common Issues and Solutions

### BFF Service Issues

#### Symptom: High latency or timeouts

```bash
# Check downstream service health
curl -f https://auth.lilith.com/v1/health
curl -f https://conversation.lilith.com/v1/health
curl -f https://ai.lilith.com/v1/health

# Check pod resources
kubectl top pods -n lilith-bff
kubectl describe pods -n lilith-bff

# Check recent deployments
kubectl rollout history deployment/bff-deployment -n lilith-bff
```

**Resolution**: Scale up pods or rollback deployment if recent change

### Authentication Service Issues

#### Symptom: Login failures

```bash
# Check JWT token validation
kubectl logs deployment/auth-deployment -n lilith-auth | grep -i "jwt\|token\|auth"

# Check vault connectivity
kubectl exec -it auth-pod -n lilith-auth -- curl -f https://vault.lilith.com/v1/sys/health

# Verify database connectivity
kubectl exec -it auth-pod -n lilith-auth -- nc -z postgres-service 5432
```

**Resolution**: Restart auth pods, verify vault secrets, check database
connectivity

### AI Service Issues

#### Symptom: Model inference failures

```bash
# Check AI service logs
kubectl logs deployment/ai-deployment -n lilith-ai | grep -i "error\|timeout\|model"

# Check GPU resources
kubectl get nodes -o wide
kubectl describe nodes | grep -A5 "nvidia.com/gpu"

# Check model provider connectivity
kubectl exec -it ai-pod -n lilith-ai -- curl -f https://api.openai.com/v1/models
```

**Resolution**: Scale GPU nodes, restart model containers, check provider API
keys

### Database Issues

#### Symptom: Connection pool exhaustion

```bash
# Check active connections
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"

# Check for blocked queries
kubectl exec -it postgres-primary-0 -n database -- psql -U postgres -c "SELECT blocked_locks.pid AS blocked_pid, blocking_locks.pid AS blocking_pid, blocked_activity.usename AS blocked_user, blocking_activity.usename AS blocking_user FROM pg_catalog.pg_locks blocked_locks JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_catalog.pg_locks blocking_locks ON(blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid AND blocking_locks.pid != blocked_locks.pid) JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.GRANTED;"
```

**Resolution**: Kill long-running queries, scale database resources, adjust
connection pool settings

## Escalation Procedures

### When to Escalate

1. **Immediate Escalation (P0)**
   - Unable to resolve within 15 minutes
   - Multiple services failing simultaneously
   - Security incidents
   - Data loss confirmed

2. **Timed Escalation (P1)**
   - No progress after 1 hour
   - Customer complaints escalating
   - Business-critical features affected

3. **Business Hours Escalation (P2/P3)**
   - Requires architecture changes
   - Multiple team coordination needed
   - Business decision required

### Escalation Contacts

```
Platform Lead: @platform-lead (primary), @platform-deputy (secondary)
Security Lead: @security-lead (immediate for security incidents)
Engineering Director: @engineering-director (for P0 after 30 minutes)
```

## Incident Communications

### Internal Updates (Every 30 minutes during P0/P1)

```
**Status Update - [Timestamp]**
Incident: [Brief description]
Impact: [User-facing impact]
Progress: [Current investigation/fix status]
ETA: [Estimated resolution time]
Next Update: [Time of next update]
```

### Customer Communications

**P0/P1 Incidents**: Update status page within 15 minutes **P2 Incidents**:
Update status page within 2 hours **P3 Incidents**: No customer communication
required

### Status Page Template

```
We are currently experiencing [issue description].
Our team is actively investigating and working on a resolution.
Affected services: [service list]
We will provide updates every [frequency] until resolved.
```

## Post-Incident Actions

### Immediate (Within 24 hours)

- [ ] Verify full service restoration
- [ ] Document timeline in incident log
- [ ] Preserve relevant logs and metrics
- [ ] Brief next on-call engineer
- [ ] Update monitoring if needed

### Follow-up (Within 72 hours)

- [ ] Complete post-incident review
- [ ] Identify root cause
- [ ] Document lessons learned
- [ ] Update runbooks if needed
- [ ] Schedule follow-up actions

## Handoff Procedures

### End of Shift

1. Review active alerts and incidents
2. Document any ongoing investigations
3. Brief incoming on-call engineer
4. Update handoff notes in shared document
5. Confirm new engineer has access to all tools

### Handoff Checklist Template

```
**On-Call Handoff - [Date/Time]**

Active Incidents:
- [List any ongoing incidents]

Ongoing Investigations:
- [List any unresolved alerts or issues]

Recent Changes:
- [Note any recent deployments or changes]

Monitoring Notes:
- [Any trending issues or capacity concerns]

Action Items:
- [Any items requiring follow-up]

Contact Info:
- Outgoing: [your contact]
- Incoming: [incoming contact]
```

## Reference Information

### Key Metrics and Thresholds

| Metric               | Warning     | Critical    | Action                  |
| -------------------- | ----------- | ----------- | ----------------------- |
| CPU Usage            | >70%        | >90%        | Scale pods              |
| Memory Usage         | >80%        | >95%        | Scale pods              |
| Disk Usage           | >80%        | >90%        | Clean up or scale       |
| Response Time        | >2s         | >5s         | Investigate/scale       |
| Error Rate           | >2%         | >5%         | Immediate investigation |
| Database Connections | >80% of max | >95% of max | Scale/optimize          |

### Useful Commands Reference

```bash
# Quick service status
kubectl get pods --all-namespaces | grep -v Running

# Resource usage across cluster
kubectl top nodes && kubectl top pods --all-namespaces --sort-by=cpu

# Recent events
kubectl get events --sort-by='.lastTimestamp' --all-namespaces

# Deployment status
kubectl get deployments --all-namespaces

# Service endpoints
kubectl get svc --all-namespaces

# Check pod logs for errors
kubectl logs -l app=<service> --tail=50 | grep -i error

# Scale service
kubectl scale deployment <deployment> --replicas=<count> -n <namespace>

# Rollback deployment
kubectl rollout undo deployment/<deployment> -n <namespace>
```

### Emergency Procedures Summary

1. **Service Down**: Check health endpoints → Check pods → Check resources →
   Scale or restart
2. **High Latency**: Check metrics → Check downstream dependencies → Scale if
   needed
3. **Database Issues**: Check connections → Check queries → Scale or optimize
4. **Security Alerts**: Isolate affected systems → Notify security team →
   Preserve evidence
5. **Deployment Issues**: Check ArgoCD → Verify manifest → Rollback if needed

---

**Remember**: When in doubt, escalate early. It's better to involve additional
help than to let an incident grow in severity.
