# SLO Alert Runbooks

This document provides runbooks for all SLO-related alerts in the Lilith
platform.

## Overview

SLO (Service Level Objective) alerts are triggered when services are consuming
their error budget too quickly or when error budgets are running low. These
alerts use multi-window, multi-burn-rate methodology to detect issues at
different timescales.

## Alert Index

| Alert                                       | Severity         | Description                                       |
| ------------------------------------------- | ---------------- | ------------------------------------------------- |
| [SLOHighBurnRate](#slohighburnrate)         | Critical         | Fast burn rate consuming 2% budget in 1 hour      |
| [SLOSlowBurnRate](#sloslowburnrate)         | Warning          | Elevated burn rate consuming 5% budget in 6 hours |
| [SLOErrorBudgetLow](#sloerrorbudgetlow)     | Warning/Critical | Error budget below threshold                      |
| [SLOLatencyViolation](#slolatencyviolation) | Warning          | Latency exceeding SLO target                      |

---

## SLOHighBurnRate

### Description

The service is consuming error budget at an extremely high rate. At the current
rate, the monthly error budget will be exhausted within 1 hour. This requires
immediate attention.

### Severity

**Critical** - Page on-call immediately

### Trigger Condition

```promql
slo:burn_rate:1h{slo_name="<service>"} > 14.4
AND
slo:burn_rate:6h{slo_name="<service>"} > 6
```

### Impact

- Users are experiencing significant service degradation
- Error budget being consumed 14.4x faster than sustainable
- If unchecked, will breach SLO within the budget window

### Investigation Steps

1. **Identify the affected service**

   ```bash
   # Check which service is alerting
   kubectl get prometheusrules -n monitoring -o yaml | grep -A5 "SLOHighBurnRate"
   ```

2. **Check current error rates**

   ```promql
   # View current error rate
   sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
   /
   sum(rate(http_requests_total{service="<service>"}[5m]))
   ```

3. **Review recent deployments**

   ```bash
   # Check recent deployments
   kubectl rollout history deployment/<service> -n lilith

   # Check pod events
   kubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service>
   ```

4. **Check pod health**

   ```bash
   kubectl get pods -n lilith -l app=<service>
   kubectl describe pods -n lilith -l app=<service>
   kubectl logs -n lilith -l app=<service> --tail=100
   ```

5. **Check dependent services**

   ```bash
   # Review dependency health in Grafana
   # Dashboard: Lilith - Service Dependencies
   ```

6. **Check infrastructure**

   ```bash
   # Node status
   kubectl get nodes

   # Resource pressure
   kubectl top nodes
   kubectl top pods -n lilith
   ```

### Remediation Actions

1. **If caused by a recent deployment:**

   ```bash
   # Rollback to previous version
   kubectl rollout undo deployment/<service> -n lilith

   # Verify rollback
   kubectl rollout status deployment/<service> -n lilith
   ```

2. **If caused by resource exhaustion:**

   ```bash
   # Scale up the deployment
   kubectl scale deployment/<service> -n lilith --replicas=<n+2>

   # Or enable HPA if not already
   kubectl apply -f deploy/k8s/<service>/hpa.yaml
   ```

3. **If caused by dependency failure:**
   - Check dependent service status
   - Enable circuit breaker if available
   - Consider graceful degradation

4. **If caused by database issues:**
   - Check RDS metrics in CloudWatch
   - Check connection pool status
   - Consider read replica failover

### Escalation Path

1. On-call SRE (immediate)
2. Service team lead (if not resolved in 15 minutes)
3. Platform team (if infrastructure-related)
4. Database team (if database-related)

### Post-Incident

- Create incident report
- Update SLO if target was unrealistic
- Add monitoring for root cause
- Review deployment process if deployment-related

---

## SLOSlowBurnRate

### Description

The service is consuming error budget at an elevated but not critical rate. At
the current rate, the monthly error budget may be exhausted within the SLO
window.

### Severity

**Warning** - Investigate within 30 minutes

### Trigger Condition

```promql
slo:burn_rate:6h{slo_name="<service>"} > 6
AND
slo:burn_rate:24h{slo_name="<service>"} > 3
```

### Impact

- Service quality is degraded
- Error budget being consumed 6x faster than sustainable
- Will likely breach SLO if trend continues

### Investigation Steps

1. **Check the trend**

   ```promql
   # View burn rate trend
   slo:burn_rate:6h{slo_name="<service>"}[24h]
   ```

2. **Identify error patterns**

   ```promql
   # Errors by endpoint
   topk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[1h])) by (handler))
   ```

3. **Check for gradual degradation**
   - Review Grafana SLO dashboard
   - Look for patterns (time-based, load-based)
   - Compare with baseline metrics

4. **Review logs for recurring errors**
   ```bash
   # Aggregate error logs
   kubectl logs -n lilith -l app=<service> --since=6h | grep -i error | sort | uniq -c | sort -rn | head -20
   ```

### Remediation Actions

1. **If gradual degradation due to load:**
   - Increase replica count
   - Review and optimize slow endpoints
   - Enable or adjust autoscaling

2. **If intermittent dependency issues:**
   - Review dependency health
   - Increase timeout/retry settings
   - Add circuit breaker protection

3. **If data-related:**
   - Check for data growth issues
   - Review query performance
   - Consider caching strategies

### Escalation Path

1. Service team (primary owner)
2. On-call SRE (if infrastructure-related)
3. Platform team (if cross-cutting)

---

## SLOErrorBudgetLow

### Description

The cumulative error budget for the service has dropped below a critical
threshold. This indicates sustained reliability issues over the SLO window.

### Severity

- **Warning** - Budget below 50%
- **Critical** - Budget below 25%

### Trigger Condition

```promql
slo:error_budget_remaining:ratio{slo_name="<service>"} < <threshold>
```

### Impact

- Limited remaining capacity for errors
- May need to freeze deployments
- Risk of SLO breach

### Investigation Steps

1. **Check current budget status**

   ```promql
   # Current error budget remaining
   slo:error_budget_remaining:ratio{slo_name="<service>"}
   ```

2. **Review budget consumption history**
   - Check SLO dashboard for consumption pattern
   - Identify major incidents that consumed budget
   - Review if any single event dominated

3. **Assess remaining window**
   ```promql
   # Days remaining in SLO window
   # Calculate based on current date vs window start
   ```

### Remediation Actions

1. **Implement deployment freeze:**
   - Pause non-critical deployments
   - Require additional review for changes
   - Focus on reliability improvements

2. **Prioritize reliability work:**
   - Address known stability issues
   - Improve error handling
   - Add resilience patterns

3. **If budget approaches zero:**
   - Consider SLO revision
   - Communicate with stakeholders
   - Document lessons learned

### Escalation Path

1. Service team lead
2. Engineering manager
3. Product manager (for stakeholder communication)

---

## SLOLatencyViolation

### Description

The service latency (at the specified percentile) is exceeding the SLO target.

### Severity

**Warning** - Investigate within 1 hour

### Trigger Condition

```promql
slo:sli_latency:seconds_rate5m{slo_name="<service>"} > <target_ms>/1000
```

### Impact

- User experience degradation
- Potential timeout cascade
- Error budget consumption

### Investigation Steps

1. **Identify slow endpoints**

   ```promql
   # P95 latency by endpoint
   histogram_quantile(0.95,
     sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m]))
     by (le, handler)
   )
   ```

2. **Check for load correlation**

   ```promql
   # Request rate vs latency correlation
   rate(http_requests_total{service="<service>"}[5m])
   ```

3. **Review traces**
   - Open Tempo/Jaeger
   - Filter for slow traces (> target latency)
   - Identify bottleneck spans

4. **Check database latency**

   ```promql
   # Database query latency
   histogram_quantile(0.95,
     sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m]))
     by (le, query_type)
   )
   ```

5. **Check external dependencies**
   ```promql
   # External call latency
   histogram_quantile(0.95,
     sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m]))
     by (le, target_service)
   )
   ```

### Remediation Actions

1. **If database-related:**
   - Add/optimize indexes
   - Enable query caching
   - Scale read replicas

2. **If resource-constrained:**
   - Scale up pods
   - Increase CPU/memory limits
   - Review for memory leaks

3. **If dependency-related:**
   - Add response caching
   - Implement async processing
   - Review timeout settings

4. **If code-related:**
   - Profile slow endpoints
   - Optimize hot paths
   - Add caching layers

### Escalation Path

1. Service team
2. Database team (if DB-related)
3. Platform team (if infrastructure-related)

---

## General Troubleshooting Tools

### Grafana Dashboards

- **SLO Overview**: `https://grafana.lilith.app/d/slo-overview`
- **Service Metrics**: `https://grafana.lilith.app/d/service-metrics`
- **Infrastructure**: `https://grafana.lilith.app/d/kubernetes-cluster`

### Useful Commands

```bash
# Quick service health check
kubectl get pods,svc,hpa -n lilith -l app=<service>

# Recent events
kubectl get events -n lilith --sort-by='.lastTimestamp' | head -30

# Pod resource usage
kubectl top pods -n lilith -l app=<service>

# Recent logs with errors
kubectl logs -n lilith -l app=<service> --since=10m | grep -i error
```

### PromQL Queries

```promql
# Current availability
sum(rate(http_requests_total{code!~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))

# Error rate
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m]))

# P95 latency
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m]))
  by (le)
)
```

## Contact Information

| Role          | Contact                |
| ------------- | ---------------------- |
| On-Call SRE   | #sre-oncall (Slack)    |
| Platform Team | #platform-team (Slack) |
| Database Team | #database-team (Slack) |
| Security Team | #security-team (Slack) |

## References

- [SLO Policy Document](../observability/slo-policies.md)
- [Incident Response Process](../incidents/incident-response.md)
- [On-Call Handbook](../on-call/handbook.md)
