# Alert Runbooks

This directory contains runbooks for responding to alerts in the Lilith
platform.

## Quick Reference

| Category                                            | Document                           | Common Alerts                         |
| --------------------------------------------------- | ---------------------------------- | ------------------------------------- |
| [SLO Alerts](./slo-alerts.md)                       | Service Level Objective violations | SLOHighBurnRate, SLOErrorBudgetLow    |
| [Infrastructure Alerts](./infrastructure-alerts.md) | Kubernetes and node issues         | KubePodCrashLooping, KubeNodeNotReady |
| [Application Alerts](./application-alerts.md)       | Service-specific issues            | HighErrorRate, ServiceDown            |

## Alert Response Workflow

1. **Acknowledge** - Acknowledge the alert in PagerDuty/Slack
2. **Assess** - Determine severity and impact
3. **Investigate** - Follow runbook investigation steps
4. **Remediate** - Apply appropriate fixes
5. **Communicate** - Update stakeholders
6. **Document** - Create incident report if needed

## Severity Levels

| Severity | Response Time     | Who to Page  |
| -------- | ----------------- | ------------ |
| Critical | Immediate         | On-call SRE  |
| Warning  | 30 minutes        | Service team |
| Info     | Next business day | Service team |

## Alert Dashboards

- **SLO Overview**: [Grafana](https://grafana.lilith.app/d/slo-overview)
- **Service Metrics**: [Grafana](https://grafana.lilith.app/d/service-metrics)
- **Infrastructure**: [Grafana](https://grafana.lilith.app/d/kubernetes-cluster)

## On-Call Contacts

- **Slack**: #sre-oncall
- **PagerDuty**: [Lilith Platform](https://lilith.pagerduty.com)

## Related Documentation

- [On-Call Handbook](../on-call/handbook.md)
- [Incident Response Process](../incidents/incident-response.md)
- [Alerting Procedures](../../observability/alerting-procedures.md)
