Eve library

SLO Alert Runbooks

Runbook

Revision 1

On this page

This document provides runbooks for all SLO-related alerts in the Lilith platform.

Overview#

SLO (Service Level Objective) alerts are triggered when services are consuming their error budget too quickly or when error budgets are running low. These alerts use multi-window, multi-burn-rate methodology to detect issues at different timescales.

Alert Index#

Alert Severity Description
SLOHighBurnRate Critical Fast burn rate consuming 2% budget in 1 hour
SLOSlowBurnRate Warning Elevated burn rate consuming 5% budget in 6 hours
SLOErrorBudgetLow Warning/Critical Error budget below threshold
SLOLatencyViolation Warning Latency exceeding SLO target

SLOHighBurnRate#

Description#

The service is consuming error budget at an extremely high rate. At the current rate, the monthly error budget will be exhausted within 1 hour. This requires immediate attention.

Severity#

Critical - Page on-call immediately

Trigger Condition#

promql
slo:burn_rate:1h{slo_name="<service>"} > 14.4
AND
slo:burn_rate:6h{slo_name="<service>"} > 6

Impact#

  • Users are experiencing significant service degradation
  • Error budget being consumed 14.4x faster than sustainable
  • If unchecked, will breach SLO within the budget window

Investigation Steps#

  1. Identify the affected service

    bash
    # Check which service is alerting
    kubectl get prometheusrules -n monitoring -o yaml | grep -A5 "SLOHighBurnRate"
    
  2. Check current error rates

    promql
    # View current error rate
    sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
    /
    sum(rate(http_requests_total{service="<service>"}[5m]))
    
  3. Review recent deployments

    bash
    # Check recent deployments
    kubectl rollout history deployment/<service> -n lilith
    
    # Check pod events
    kubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service>
    
  4. Check pod health

    bash
    kubectl get pods -n lilith -l app=<service>
    kubectl describe pods -n lilith -l app=<service>
    kubectl logs -n lilith -l app=<service> --tail=100
    
  5. Check dependent services

    bash
    # Review dependency health in Grafana
    # Dashboard: Lilith - Service Dependencies
    
  6. Check infrastructure

    bash
    # Node status
    kubectl get nodes
    
    # Resource pressure
    kubectl top nodes
    kubectl top pods -n lilith
    

Remediation Actions#

  1. If caused by a recent deployment:

    bash
    # Rollback to previous version
    kubectl rollout undo deployment/<service> -n lilith
    
    # Verify rollback
    kubectl rollout status deployment/<service> -n lilith
    
  2. If caused by resource exhaustion:

    bash
    # Scale up the deployment
    kubectl scale deployment/<service> -n lilith --replicas=<n+2>
    
    # Or enable HPA if not already
    kubectl apply -f deploy/k8s/<service>/hpa.yaml
    
  3. If caused by dependency failure:

    • Check dependent service status
    • Enable circuit breaker if available
    • Consider graceful degradation
  4. If caused by database issues:

    • Check RDS metrics in CloudWatch
    • Check connection pool status
    • Consider read replica failover

Escalation Path#

  1. On-call SRE (immediate)
  2. Service team lead (if not resolved in 15 minutes)
  3. Platform team (if infrastructure-related)
  4. Database team (if database-related)

Post-Incident#

  • Create incident report
  • Update SLO if target was unrealistic
  • Add monitoring for root cause
  • Review deployment process if deployment-related

SLOSlowBurnRate#

Description#

The service is consuming error budget at an elevated but not critical rate. At the current rate, the monthly error budget may be exhausted within the SLO window.

Severity#

Warning - Investigate within 30 minutes

Trigger Condition#

promql
slo:burn_rate:6h{slo_name="<service>"} > 6
AND
slo:burn_rate:24h{slo_name="<service>"} > 3

Impact#

  • Service quality is degraded
  • Error budget being consumed 6x faster than sustainable
  • Will likely breach SLO if trend continues

Investigation Steps#

  1. Check the trend

    promql
    # View burn rate trend
    slo:burn_rate:6h{slo_name="<service>"}[24h]
    
  2. Identify error patterns

    promql
    # Errors by endpoint
    topk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[1h])) by (handler))
    
  3. Check for gradual degradation

    • Review Grafana SLO dashboard
    • Look for patterns (time-based, load-based)
    • Compare with baseline metrics
  4. Review logs for recurring errors

    bash
    # Aggregate error logs
    kubectl logs -n lilith -l app=<service> --since=6h | grep -i error | sort | uniq -c | sort -rn | head -20
    

Remediation Actions#

  1. If gradual degradation due to load:

    • Increase replica count
    • Review and optimize slow endpoints
    • Enable or adjust autoscaling
  2. If intermittent dependency issues:

    • Review dependency health
    • Increase timeout/retry settings
    • Add circuit breaker protection
  3. If data-related:

    • Check for data growth issues
    • Review query performance
    • Consider caching strategies

Escalation Path#

  1. Service team (primary owner)
  2. On-call SRE (if infrastructure-related)
  3. Platform team (if cross-cutting)

SLOErrorBudgetLow#

Description#

The cumulative error budget for the service has dropped below a critical threshold. This indicates sustained reliability issues over the SLO window.

Severity#

  • Warning - Budget below 50%
  • Critical - Budget below 25%

Trigger Condition#

promql
slo:error_budget_remaining:ratio{slo_name="<service>"} < <threshold>

Impact#

  • Limited remaining capacity for errors
  • May need to freeze deployments
  • Risk of SLO breach

Investigation Steps#

  1. Check current budget status

    promql
    # Current error budget remaining
    slo:error_budget_remaining:ratio{slo_name="<service>"}
    
  2. Review budget consumption history

    • Check SLO dashboard for consumption pattern
    • Identify major incidents that consumed budget
    • Review if any single event dominated
  3. Assess remaining window

    promql
    # Days remaining in SLO window
    # Calculate based on current date vs window start
    

Remediation Actions#

  1. Implement deployment freeze:

    • Pause non-critical deployments
    • Require additional review for changes
    • Focus on reliability improvements
  2. Prioritize reliability work:

    • Address known stability issues
    • Improve error handling
    • Add resilience patterns
  3. If budget approaches zero:

    • Consider SLO revision
    • Communicate with stakeholders
    • Document lessons learned

Escalation Path#

  1. Service team lead
  2. Engineering manager
  3. Product manager (for stakeholder communication)

SLOLatencyViolation#

Description#

The service latency (at the specified percentile) is exceeding the SLO target.

Severity#

Warning - Investigate within 1 hour

Trigger Condition#

promql
slo:sli_latency:seconds_rate5m{slo_name="<service>"} > <target_ms>/1000

Impact#

  • User experience degradation
  • Potential timeout cascade
  • Error budget consumption

Investigation Steps#

  1. Identify slow endpoints

    promql
    # P95 latency by endpoint
    histogram_quantile(0.95,
      sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m]))
      by (le, handler)
    )
    
  2. Check for load correlation

    promql
    # Request rate vs latency correlation
    rate(http_requests_total{service="<service>"}[5m])
    
  3. Review traces

    • Open Tempo/Jaeger
    • Filter for slow traces (> target latency)
    • Identify bottleneck spans
  4. Check database latency

    promql
    # Database query latency
    histogram_quantile(0.95,
      sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m]))
      by (le, query_type)
    )
    
  5. Check external dependencies

    promql
    # External call latency
    histogram_quantile(0.95,
      sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m]))
      by (le, target_service)
    )
    

Remediation Actions#

  1. If database-related:

    • Add/optimize indexes
    • Enable query caching
    • Scale read replicas
  2. If resource-constrained:

    • Scale up pods
    • Increase CPU/memory limits
    • Review for memory leaks
  3. If dependency-related:

    • Add response caching
    • Implement async processing
    • Review timeout settings
  4. If code-related:

    • Profile slow endpoints
    • Optimize hot paths
    • Add caching layers

Escalation Path#

  1. Service team
  2. Database team (if DB-related)
  3. Platform team (if infrastructure-related)

General Troubleshooting Tools#

Grafana Dashboards#

  • SLO Overview: https://grafana.lilith.app/d/slo-overview
  • Service Metrics: https://grafana.lilith.app/d/service-metrics
  • Infrastructure: https://grafana.lilith.app/d/kubernetes-cluster

Useful Commands#

bash
# Quick service health check
kubectl get pods,svc,hpa -n lilith -l app=<service>

# Recent events
kubectl get events -n lilith --sort-by='.lastTimestamp' | head -30

# Pod resource usage
kubectl top pods -n lilith -l app=<service>

# Recent logs with errors
kubectl logs -n lilith -l app=<service> --since=10m | grep -i error

PromQL Queries#

promql
# Current availability
sum(rate(http_requests_total{code!~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))

# Error rate
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))

# P95 latency
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m]))
  by (le)
)

Contact Information#

Role Contact
On-Call SRE #sre-oncall (Slack)
Platform Team #platform-team (Slack)
Database Team #database-team (Slack)
Security Team #security-team (Slack)

References#