This document provides runbooks for application-specific alerts in the Lilith platform.
Overview#
Application alerts monitor the health and performance of Lilith services including auth, content, conversation, media, notification, and AI services.
Alert Index#
| Alert | Severity | Description |
|---|---|---|
| ServiceDown | Critical | Service not responding |
| HighErrorRate | Critical/Warning | High percentage of failed requests |
| HighLatency | Warning | Response times exceeding thresholds |
| DatabaseConnectionFailure | Critical | Cannot connect to database |
| QueueBacklog | Warning | Message queue backlog growing |
| RateLimitExceeded | Warning | Rate limiting being triggered |
| AuthenticationFailures | Warning | Unusual authentication failures |
| AIServiceDegraded | Warning | AI service performance degraded |
| MediaProcessingBacklog | Warning | Media processing queue growing |
| NotificationDeliveryFailure | Warning | Notifications failing to deliver |
ServiceDown#
Description#
A service is not responding to health checks and is considered down.
Severity#
Critical - Immediate attention required
Trigger Condition#
up{job=~"lilith-.*"} == 0
Impact#
- Complete service outage
- User-facing errors
- Potential data loss for in-flight requests
Investigation Steps#
-
Check pod status
bashkubectl get pods -n lilith -l app=<service> kubectl describe pods -n lilith -l app=<service> -
Check recent events
bashkubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service> -
Check logs
bashkubectl logs -n lilith -l app=<service> --tail=100 kubectl logs -n lilith -l app=<service> --previous --tail=100 -
Check health endpoint
bashkubectl exec -it <any-pod> -n lilith -- curl http://<service>:8080/health -
Check dependencies
- Database connectivity
- Cache availability
- External service access
Remediation Actions#
-
If pods are crashing:
bash# Check crash reason and follow KubePodCrashLooping runbook kubectl logs <pod> -n lilith --previous -
If pods are pending:
bash# Check scheduling issues kubectl describe pod <pod> -n lilith | grep -A20 Events -
If networking issue:
bash# Check service endpoints kubectl get endpoints <service> -n lilith kubectl get svc <service> -n lilith -
Restart deployment:
bashkubectl rollout restart deployment/<service> -n lilith kubectl rollout status deployment/<service> -n lilith
Escalation Path#
- On-call SRE
- Service team lead
- Platform team
HighErrorRate#
Description#
The service is returning a high percentage of error responses (5xx status codes).
Severity#
- Critical - Error rate > 10%
- Warning - Error rate > 1%
Trigger Condition#
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m])) > 0.01
Investigation Steps#
-
Identify error types
promqlsum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (code) -
Find affected endpoints
promqltopk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (handler)) -
Check logs for errors
bashkubectl logs -n lilith -l app=<service> --tail=500 | grep -i "error\|exception\|fatal" -
Check traces for failures
- Open Tempo/Jaeger
- Filter for error spans
- Identify root cause span
-
Check dependency health
bashkubectl get pods -n lilith # Check database, cache, external services
Remediation Actions#
-
If dependency failure:
- Address dependency issue first
- Enable circuit breaker if available
- Consider graceful degradation
-
If code bug:
bash# Rollback to last known good version kubectl rollout undo deployment/<service> -n lilith -
If resource exhaustion:
bash# Scale up kubectl scale deployment/<service> -n lilith --replicas=<n+2> -
If rate limit related:
- Increase rate limits temporarily
- Add caching to reduce load
HighLatency#
Description#
Service response times are exceeding acceptable thresholds.
Severity#
Warning - Investigate within 30 minutes
Trigger Condition#
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le)) > 1
Investigation Steps#
-
Identify slow endpoints
promqltopk(10, histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le, handler))) -
Check database performance
promqlhistogram_quantile(0.95, sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m])) by (le, query)) -
Check external call latency
promqlhistogram_quantile(0.95, sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m])) by (le, target)) -
Review traces
- Open Tempo/Jaeger
- Filter for slow traces
- Identify slowest span
-
Check resource usage
bashkubectl top pods -n lilith -l app=<service>
Remediation Actions#
-
If database slow:
- Check for missing indexes
- Review query plans
- Add read replica
-
If resource constrained:
bashkubectl scale deployment/<service> -n lilith --replicas=<n+2> -
If external dependency slow:
- Add caching
- Increase timeout
- Consider async processing
-
If N+1 queries:
- Review code for batch loading
- Add eager loading
DatabaseConnectionFailure#
Description#
Service cannot establish connections to the database.
Severity#
Critical - Immediate attention required
Trigger Condition#
sum(db_connection_errors_total{service="<service>"}) > 0
Investigation Steps#
-
Check database status
bash# Check RDS console or aws rds describe-db-instances --db-instance-identifier <instance-id> -
Check connection pool
promqldb_pool_connections_active{service="<service>"} db_pool_connections_idle{service="<service>"} -
Check network connectivity
bashkubectl exec -it <pod> -n lilith -- nc -zv <db-host> 5432 -
Check credentials
bashkubectl get secret <db-secret> -n lilith -o yaml
Remediation Actions#
-
If database down:
- Check RDS events in AWS console
- Initiate failover if needed
-
If connection exhaustion:
- Increase connection pool size
- Check for connection leaks
- Scale down pods to reduce pressure
-
If network issue:
- Check security groups
- Verify VPC peering
- Check DNS resolution
QueueBacklog#
Description#
Message queue (SQS, RabbitMQ, etc.) has a growing backlog of unprocessed messages.
Severity#
Warning - Investigate within 30 minutes
Trigger Condition#
queue_messages_pending{queue="<queue-name>"} > 1000
Investigation Steps#
-
Check queue metrics
promqlqueue_messages_pending{queue="<queue-name>"} rate(queue_messages_processed_total{queue="<queue-name>"}[5m]) -
Check consumer health
bashkubectl get pods -n lilith -l app=<consumer-service> kubectl logs -n lilith -l app=<consumer-service> --tail=100 -
Check for processing errors
promqlrate(queue_messages_failed_total{queue="<queue-name>"}[5m])
Remediation Actions#
-
Scale up consumers:
bashkubectl scale deployment/<consumer> -n lilith --replicas=<n+2> -
If consumer failing:
- Fix consumer issue first
- Consider DLQ review
-
If producer spike:
- Verify traffic patterns
- Consider rate limiting producers
RateLimitExceeded#
Description#
API rate limits are being triggered, indicating high traffic or potential abuse.
Severity#
Warning - Monitor and investigate
Trigger Condition#
sum(rate(http_requests_total{code="429"}[5m])) > 10
Investigation Steps#
-
Identify affected clients
promqltopk(10, sum(rate(http_requests_total{code="429"}[5m])) by (client_id)) -
Check traffic patterns
promqlsum(rate(http_requests_total{service="<service>"}[5m])) by (handler) -
Review for abuse patterns
- Check client IPs
- Review request patterns
- Look for bot signatures
Remediation Actions#
-
If legitimate traffic:
- Increase rate limits
- Add caching
- Scale infrastructure
-
If abuse:
- Block offending IPs/clients
- Add additional security measures
- Report to security team
AuthenticationFailures#
Description#
Unusual number of authentication failures detected.
Severity#
Warning - May indicate attack or misconfiguration
Trigger Condition#
sum(rate(lilith_auth_failures_total[5m])) > 100
Investigation Steps#
-
Check failure types
promqlsum(rate(lilith_auth_failures_total[5m])) by (reason) -
Identify sources
promqltopk(10, sum(rate(lilith_auth_failures_total[5m])) by (client_ip)) -
Check for patterns
- Same user, multiple attempts
- Same IP, different users
- Geographic anomalies
Remediation Actions#
-
If brute force attack:
- Block offending IPs
- Enable additional rate limiting
- Alert security team
-
If misconfiguration:
- Check OAuth/JWT settings
- Verify secret rotation
- Review recent changes
AIServiceDegraded#
Description#
AI service (LLM, embeddings, etc.) is experiencing degraded performance.
Severity#
Warning - User experience impact
Trigger Condition#
histogram_quantile(0.95, sum(rate(lilith_ai_request_duration_seconds_bucket[5m])) by (le)) > 30
Investigation Steps#
-
Check AI provider status
- OpenAI/Anthropic status pages
- Check for announced outages
-
Check queue depth
promqllilith_ai_queue_depth -
Check token usage
promqlsum(rate(lilith_ai_tokens_used_total[5m])) by (model)
Remediation Actions#
-
If provider issue:
- Switch to backup provider
- Enable fallback responses
- Communicate to users
-
If rate limited:
- Reduce request rate
- Enable request queuing
- Consider upgrading plan
MediaProcessingBacklog#
Description#
Media processing queue has accumulated a backlog.
Severity#
Warning - Delayed media availability
Investigation Steps#
-
Check queue status
promqllilith_media_processing_queue_depth -
Check processing times
promqlhistogram_quantile(0.95, sum(rate(lilith_media_processing_duration_seconds_bucket[5m])) by (le, type)) -
Check worker health
bashkubectl get pods -n lilith -l app=media-processor
Remediation Actions#
-
Scale up processors:
bashkubectl scale deployment/media-processor -n lilith --replicas=<n+2> -
If GPU constrained:
- Add GPU nodes
- Prioritize queue
NotificationDeliveryFailure#
Description#
Notifications (push, email, SMS) are failing to deliver.
Severity#
Warning - User communication impacted
Trigger Condition#
sum(rate(lilith_notifications_failed_total[5m])) / sum(rate(lilith_notifications_sent_total[5m])) > 0.05
Investigation Steps#
-
Check failure reasons
promqlsum(rate(lilith_notifications_failed_total[5m])) by (channel, reason) -
Check provider status
- FCM/APNs status
- Email provider status
- SMS provider status
Remediation Actions#
-
If push token issues:
- Clean up stale tokens
- Check app registration
-
If email issues:
- Check SPF/DKIM
- Review bounce rates
- Check sending limits
General Troubleshooting#
Useful PromQL Queries#
# Error rate by service
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
# Latency by service
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
# Request rate by service
sum(rate(http_requests_total[5m])) by (service)
Quick Diagnostics#
# Service health check
for svc in auth bff content conversation media notification ai; do
echo "Checking $svc..."
kubectl get pods -n lilith -l app=$svc -o wide
done
# Recent errors in all services
for svc in auth bff content conversation media notification ai; do
echo "=== $svc errors ==="
kubectl logs -n lilith -l app=$svc --since=5m | grep -i error | tail -5
done
Contact Information#
| Team | Contact | Escalation |
|---|---|---|
| Auth Team | #auth-team | auth-oncall@lilith.app |
| Content Team | #content-team | content-oncall@lilith.app |
| AI Team | #ai-team | ai-oncall@lilith.app |
| Platform Team | #platform-team | platform-oncall@lilith.app |