This document provides runbooks for all SLO-related alerts in the Lilith platform.
Overview#
SLO (Service Level Objective) alerts are triggered when services are consuming their error budget too quickly or when error budgets are running low. These alerts use multi-window, multi-burn-rate methodology to detect issues at different timescales.
Alert Index#
| Alert | Severity | Description |
|---|---|---|
| SLOHighBurnRate | Critical | Fast burn rate consuming 2% budget in 1 hour |
| SLOSlowBurnRate | Warning | Elevated burn rate consuming 5% budget in 6 hours |
| SLOErrorBudgetLow | Warning/Critical | Error budget below threshold |
| SLOLatencyViolation | Warning | Latency exceeding SLO target |
SLOHighBurnRate#
Description#
The service is consuming error budget at an extremely high rate. At the current rate, the monthly error budget will be exhausted within 1 hour. This requires immediate attention.
Severity#
Critical - Page on-call immediately
Trigger Condition#
slo:burn_rate:1h{slo_name="<service>"} > 14.4
AND
slo:burn_rate:6h{slo_name="<service>"} > 6
Impact#
- Users are experiencing significant service degradation
- Error budget being consumed 14.4x faster than sustainable
- If unchecked, will breach SLO within the budget window
Investigation Steps#
-
Identify the affected service
bash# Check which service is alerting kubectl get prometheusrules -n monitoring -o yaml | grep -A5 "SLOHighBurnRate" -
Check current error rates
promql# View current error rate sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) / sum(rate(http_requests_total{service="<service>"}[5m])) -
Review recent deployments
bash# Check recent deployments kubectl rollout history deployment/<service> -n lilith # Check pod events kubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service> -
Check pod health
bashkubectl get pods -n lilith -l app=<service> kubectl describe pods -n lilith -l app=<service> kubectl logs -n lilith -l app=<service> --tail=100 -
Check dependent services
bash# Review dependency health in Grafana # Dashboard: Lilith - Service Dependencies -
Check infrastructure
bash# Node status kubectl get nodes # Resource pressure kubectl top nodes kubectl top pods -n lilith
Remediation Actions#
-
If caused by a recent deployment:
bash# Rollback to previous version kubectl rollout undo deployment/<service> -n lilith # Verify rollback kubectl rollout status deployment/<service> -n lilith -
If caused by resource exhaustion:
bash# Scale up the deployment kubectl scale deployment/<service> -n lilith --replicas=<n+2> # Or enable HPA if not already kubectl apply -f deploy/k8s/<service>/hpa.yaml -
If caused by dependency failure:
- Check dependent service status
- Enable circuit breaker if available
- Consider graceful degradation
-
If caused by database issues:
- Check RDS metrics in CloudWatch
- Check connection pool status
- Consider read replica failover
Escalation Path#
- On-call SRE (immediate)
- Service team lead (if not resolved in 15 minutes)
- Platform team (if infrastructure-related)
- Database team (if database-related)
Post-Incident#
- Create incident report
- Update SLO if target was unrealistic
- Add monitoring for root cause
- Review deployment process if deployment-related
SLOSlowBurnRate#
Description#
The service is consuming error budget at an elevated but not critical rate. At the current rate, the monthly error budget may be exhausted within the SLO window.
Severity#
Warning - Investigate within 30 minutes
Trigger Condition#
slo:burn_rate:6h{slo_name="<service>"} > 6
AND
slo:burn_rate:24h{slo_name="<service>"} > 3
Impact#
- Service quality is degraded
- Error budget being consumed 6x faster than sustainable
- Will likely breach SLO if trend continues
Investigation Steps#
-
Check the trend
promql# View burn rate trend slo:burn_rate:6h{slo_name="<service>"}[24h] -
Identify error patterns
promql# Errors by endpoint topk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[1h])) by (handler)) -
Check for gradual degradation
- Review Grafana SLO dashboard
- Look for patterns (time-based, load-based)
- Compare with baseline metrics
-
Review logs for recurring errors
bash# Aggregate error logs kubectl logs -n lilith -l app=<service> --since=6h | grep -i error | sort | uniq -c | sort -rn | head -20
Remediation Actions#
-
If gradual degradation due to load:
- Increase replica count
- Review and optimize slow endpoints
- Enable or adjust autoscaling
-
If intermittent dependency issues:
- Review dependency health
- Increase timeout/retry settings
- Add circuit breaker protection
-
If data-related:
- Check for data growth issues
- Review query performance
- Consider caching strategies
Escalation Path#
- Service team (primary owner)
- On-call SRE (if infrastructure-related)
- Platform team (if cross-cutting)
SLOErrorBudgetLow#
Description#
The cumulative error budget for the service has dropped below a critical threshold. This indicates sustained reliability issues over the SLO window.
Severity#
- Warning - Budget below 50%
- Critical - Budget below 25%
Trigger Condition#
slo:error_budget_remaining:ratio{slo_name="<service>"} < <threshold>
Impact#
- Limited remaining capacity for errors
- May need to freeze deployments
- Risk of SLO breach
Investigation Steps#
-
Check current budget status
promql# Current error budget remaining slo:error_budget_remaining:ratio{slo_name="<service>"} -
Review budget consumption history
- Check SLO dashboard for consumption pattern
- Identify major incidents that consumed budget
- Review if any single event dominated
-
Assess remaining window
promql# Days remaining in SLO window # Calculate based on current date vs window start
Remediation Actions#
-
Implement deployment freeze:
- Pause non-critical deployments
- Require additional review for changes
- Focus on reliability improvements
-
Prioritize reliability work:
- Address known stability issues
- Improve error handling
- Add resilience patterns
-
If budget approaches zero:
- Consider SLO revision
- Communicate with stakeholders
- Document lessons learned
Escalation Path#
- Service team lead
- Engineering manager
- Product manager (for stakeholder communication)
SLOLatencyViolation#
Description#
The service latency (at the specified percentile) is exceeding the SLO target.
Severity#
Warning - Investigate within 1 hour
Trigger Condition#
slo:sli_latency:seconds_rate5m{slo_name="<service>"} > <target_ms>/1000
Impact#
- User experience degradation
- Potential timeout cascade
- Error budget consumption
Investigation Steps#
-
Identify slow endpoints
promql# P95 latency by endpoint histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le, handler) ) -
Check for load correlation
promql# Request rate vs latency correlation rate(http_requests_total{service="<service>"}[5m]) -
Review traces
- Open Tempo/Jaeger
- Filter for slow traces (> target latency)
- Identify bottleneck spans
-
Check database latency
promql# Database query latency histogram_quantile(0.95, sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m])) by (le, query_type) ) -
Check external dependencies
promql# External call latency histogram_quantile(0.95, sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m])) by (le, target_service) )
Remediation Actions#
-
If database-related:
- Add/optimize indexes
- Enable query caching
- Scale read replicas
-
If resource-constrained:
- Scale up pods
- Increase CPU/memory limits
- Review for memory leaks
-
If dependency-related:
- Add response caching
- Implement async processing
- Review timeout settings
-
If code-related:
- Profile slow endpoints
- Optimize hot paths
- Add caching layers
Escalation Path#
- Service team
- Database team (if DB-related)
- Platform team (if infrastructure-related)
General Troubleshooting Tools#
Grafana Dashboards#
- SLO Overview:
https://grafana.lilith.app/d/slo-overview - Service Metrics:
https://grafana.lilith.app/d/service-metrics - Infrastructure:
https://grafana.lilith.app/d/kubernetes-cluster
Useful Commands#
# Quick service health check
kubectl get pods,svc,hpa -n lilith -l app=<service>
# Recent events
kubectl get events -n lilith --sort-by='.lastTimestamp' | head -30
# Pod resource usage
kubectl top pods -n lilith -l app=<service>
# Recent logs with errors
kubectl logs -n lilith -l app=<service> --since=10m | grep -i error
PromQL Queries#
# Current availability
sum(rate(http_requests_total{code!~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))
# Error rate
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))
# P95 latency
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m]))
by (le)
)
Contact Information#
| Role | Contact |
|---|---|
| On-Call SRE | #sre-oncall (Slack) |
| Platform Team | #platform-team (Slack) |
| Database Team | #database-team (Slack) |
| Security Team | #security-team (Slack) |