Eve library

Application Alert Runbooks

Runbook

Revision 1

On this page

This document provides runbooks for application-specific alerts in the Lilith platform.

Overview#

Application alerts monitor the health and performance of Lilith services including auth, content, conversation, media, notification, and AI services.

Alert Index#

Alert Severity Description
ServiceDown Critical Service not responding
HighErrorRate Critical/Warning High percentage of failed requests
HighLatency Warning Response times exceeding thresholds
DatabaseConnectionFailure Critical Cannot connect to database
QueueBacklog Warning Message queue backlog growing
RateLimitExceeded Warning Rate limiting being triggered
AuthenticationFailures Warning Unusual authentication failures
AIServiceDegraded Warning AI service performance degraded
MediaProcessingBacklog Warning Media processing queue growing
NotificationDeliveryFailure Warning Notifications failing to deliver

ServiceDown#

Description#

A service is not responding to health checks and is considered down.

Severity#

Critical - Immediate attention required

Trigger Condition#

promql
up{job=~"lilith-.*"} == 0

Impact#

  • Complete service outage
  • User-facing errors
  • Potential data loss for in-flight requests

Investigation Steps#

  1. Check pod status

    bash
    kubectl get pods -n lilith -l app=<service>
    kubectl describe pods -n lilith -l app=<service>
    
  2. Check recent events

    bash
    kubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service>
    
  3. Check logs

    bash
    kubectl logs -n lilith -l app=<service> --tail=100
    kubectl logs -n lilith -l app=<service> --previous --tail=100
    
  4. Check health endpoint

    bash
    kubectl exec -it <any-pod> -n lilith -- curl http://<service>:8080/health
    
  5. Check dependencies

    • Database connectivity
    • Cache availability
    • External service access

Remediation Actions#

  1. If pods are crashing:

    bash
    # Check crash reason and follow KubePodCrashLooping runbook
    kubectl logs <pod> -n lilith --previous
    
  2. If pods are pending:

    bash
    # Check scheduling issues
    kubectl describe pod <pod> -n lilith | grep -A20 Events
    
  3. If networking issue:

    bash
    # Check service endpoints
    kubectl get endpoints <service> -n lilith
    kubectl get svc <service> -n lilith
    
  4. Restart deployment:

    bash
    kubectl rollout restart deployment/<service> -n lilith
    kubectl rollout status deployment/<service> -n lilith
    

Escalation Path#

  1. On-call SRE
  2. Service team lead
  3. Platform team

HighErrorRate#

Description#

The service is returning a high percentage of error responses (5xx status codes).

Severity#

  • Critical - Error rate > 10%
  • Warning - Error rate > 1%

Trigger Condition#

promql
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m])) > 0.01

Investigation Steps#

  1. Identify error types

    promql
    sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (code)
    
  2. Find affected endpoints

    promql
    topk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (handler))
    
  3. Check logs for errors

    bash
    kubectl logs -n lilith -l app=<service> --tail=500 | grep -i "error\|exception\|fatal"
    
  4. Check traces for failures

    • Open Tempo/Jaeger
    • Filter for error spans
    • Identify root cause span
  5. Check dependency health

    bash
    kubectl get pods -n lilith
    # Check database, cache, external services
    

Remediation Actions#

  1. If dependency failure:

    • Address dependency issue first
    • Enable circuit breaker if available
    • Consider graceful degradation
  2. If code bug:

    bash
    # Rollback to last known good version
    kubectl rollout undo deployment/<service> -n lilith
    
  3. If resource exhaustion:

    bash
    # Scale up
    kubectl scale deployment/<service> -n lilith --replicas=<n+2>
    
  4. If rate limit related:

    • Increase rate limits temporarily
    • Add caching to reduce load

HighLatency#

Description#

Service response times are exceeding acceptable thresholds.

Severity#

Warning - Investigate within 30 minutes

Trigger Condition#

promql
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le)) > 1

Investigation Steps#

  1. Identify slow endpoints

    promql
    topk(10, histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le, handler)))
    
  2. Check database performance

    promql
    histogram_quantile(0.95, sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m])) by (le, query))
    
  3. Check external call latency

    promql
    histogram_quantile(0.95, sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m])) by (le, target))
    
  4. Review traces

    • Open Tempo/Jaeger
    • Filter for slow traces
    • Identify slowest span
  5. Check resource usage

    bash
    kubectl top pods -n lilith -l app=<service>
    

Remediation Actions#

  1. If database slow:

    • Check for missing indexes
    • Review query plans
    • Add read replica
  2. If resource constrained:

    bash
    kubectl scale deployment/<service> -n lilith --replicas=<n+2>
    
  3. If external dependency slow:

    • Add caching
    • Increase timeout
    • Consider async processing
  4. If N+1 queries:

    • Review code for batch loading
    • Add eager loading

DatabaseConnectionFailure#

Description#

Service cannot establish connections to the database.

Severity#

Critical - Immediate attention required

Trigger Condition#

promql
sum(db_connection_errors_total{service="<service>"}) > 0

Investigation Steps#

  1. Check database status

    bash
    # Check RDS console or
    aws rds describe-db-instances --db-instance-identifier <instance-id>
    
  2. Check connection pool

    promql
    db_pool_connections_active{service="<service>"}
    db_pool_connections_idle{service="<service>"}
    
  3. Check network connectivity

    bash
    kubectl exec -it <pod> -n lilith -- nc -zv <db-host> 5432
    
  4. Check credentials

    bash
    kubectl get secret <db-secret> -n lilith -o yaml
    

Remediation Actions#

  1. If database down:

    • Check RDS events in AWS console
    • Initiate failover if needed
  2. If connection exhaustion:

    • Increase connection pool size
    • Check for connection leaks
    • Scale down pods to reduce pressure
  3. If network issue:

    • Check security groups
    • Verify VPC peering
    • Check DNS resolution

QueueBacklog#

Description#

Message queue (SQS, RabbitMQ, etc.) has a growing backlog of unprocessed messages.

Severity#

Warning - Investigate within 30 minutes

Trigger Condition#

promql
queue_messages_pending{queue="<queue-name>"} > 1000

Investigation Steps#

  1. Check queue metrics

    promql
    queue_messages_pending{queue="<queue-name>"}
    rate(queue_messages_processed_total{queue="<queue-name>"}[5m])
    
  2. Check consumer health

    bash
    kubectl get pods -n lilith -l app=<consumer-service>
    kubectl logs -n lilith -l app=<consumer-service> --tail=100
    
  3. Check for processing errors

    promql
    rate(queue_messages_failed_total{queue="<queue-name>"}[5m])
    

Remediation Actions#

  1. Scale up consumers:

    bash
    kubectl scale deployment/<consumer> -n lilith --replicas=<n+2>
    
  2. If consumer failing:

    • Fix consumer issue first
    • Consider DLQ review
  3. If producer spike:

    • Verify traffic patterns
    • Consider rate limiting producers

RateLimitExceeded#

Description#

API rate limits are being triggered, indicating high traffic or potential abuse.

Severity#

Warning - Monitor and investigate

Trigger Condition#

promql
sum(rate(http_requests_total{code="429"}[5m])) > 10

Investigation Steps#

  1. Identify affected clients

    promql
    topk(10, sum(rate(http_requests_total{code="429"}[5m])) by (client_id))
    
  2. Check traffic patterns

    promql
    sum(rate(http_requests_total{service="<service>"}[5m])) by (handler)
    
  3. Review for abuse patterns

    • Check client IPs
    • Review request patterns
    • Look for bot signatures

Remediation Actions#

  1. If legitimate traffic:

    • Increase rate limits
    • Add caching
    • Scale infrastructure
  2. If abuse:

    • Block offending IPs/clients
    • Add additional security measures
    • Report to security team

AuthenticationFailures#

Description#

Unusual number of authentication failures detected.

Severity#

Warning - May indicate attack or misconfiguration

Trigger Condition#

promql
sum(rate(lilith_auth_failures_total[5m])) > 100

Investigation Steps#

  1. Check failure types

    promql
    sum(rate(lilith_auth_failures_total[5m])) by (reason)
    
  2. Identify sources

    promql
    topk(10, sum(rate(lilith_auth_failures_total[5m])) by (client_ip))
    
  3. Check for patterns

    • Same user, multiple attempts
    • Same IP, different users
    • Geographic anomalies

Remediation Actions#

  1. If brute force attack:

    • Block offending IPs
    • Enable additional rate limiting
    • Alert security team
  2. If misconfiguration:

    • Check OAuth/JWT settings
    • Verify secret rotation
    • Review recent changes

AIServiceDegraded#

Description#

AI service (LLM, embeddings, etc.) is experiencing degraded performance.

Severity#

Warning - User experience impact

Trigger Condition#

promql
histogram_quantile(0.95, sum(rate(lilith_ai_request_duration_seconds_bucket[5m])) by (le)) > 30

Investigation Steps#

  1. Check AI provider status

    • OpenAI/Anthropic status pages
    • Check for announced outages
  2. Check queue depth

    promql
    lilith_ai_queue_depth
    
  3. Check token usage

    promql
    sum(rate(lilith_ai_tokens_used_total[5m])) by (model)
    

Remediation Actions#

  1. If provider issue:

    • Switch to backup provider
    • Enable fallback responses
    • Communicate to users
  2. If rate limited:

    • Reduce request rate
    • Enable request queuing
    • Consider upgrading plan

MediaProcessingBacklog#

Description#

Media processing queue has accumulated a backlog.

Severity#

Warning - Delayed media availability

Investigation Steps#

  1. Check queue status

    promql
    lilith_media_processing_queue_depth
    
  2. Check processing times

    promql
    histogram_quantile(0.95, sum(rate(lilith_media_processing_duration_seconds_bucket[5m])) by (le, type))
    
  3. Check worker health

    bash
    kubectl get pods -n lilith -l app=media-processor
    

Remediation Actions#

  1. Scale up processors:

    bash
    kubectl scale deployment/media-processor -n lilith --replicas=<n+2>
    
  2. If GPU constrained:

    • Add GPU nodes
    • Prioritize queue

NotificationDeliveryFailure#

Description#

Notifications (push, email, SMS) are failing to deliver.

Severity#

Warning - User communication impacted

Trigger Condition#

promql
sum(rate(lilith_notifications_failed_total[5m])) / sum(rate(lilith_notifications_sent_total[5m])) > 0.05

Investigation Steps#

  1. Check failure reasons

    promql
    sum(rate(lilith_notifications_failed_total[5m])) by (channel, reason)
    
  2. Check provider status

    • FCM/APNs status
    • Email provider status
    • SMS provider status

Remediation Actions#

  1. If push token issues:

    • Clean up stale tokens
    • Check app registration
  2. If email issues:

    • Check SPF/DKIM
    • Review bounce rates
    • Check sending limits

General Troubleshooting#

Useful PromQL Queries#

promql
# Error rate by service
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Latency by service
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Request rate by service
sum(rate(http_requests_total[5m])) by (service)

Quick Diagnostics#

bash
# Service health check
for svc in auth bff content conversation media notification ai; do
  echo "Checking $svc..."
  kubectl get pods -n lilith -l app=$svc -o wide
done

# Recent errors in all services
for svc in auth bff content conversation media notification ai; do
  echo "=== $svc errors ==="
  kubectl logs -n lilith -l app=$svc --since=5m | grep -i error | tail -5
done

Contact Information#

Team Contact Escalation
Auth Team #auth-team auth-oncall@lilith.app
Content Team #content-team content-oncall@lilith.app
AI Team #ai-team ai-oncall@lilith.app
Platform Team #platform-team platform-oncall@lilith.app

References#