Eve library
Alert Runbooks
Runbook
Revision 1
On this page
This directory contains runbooks for responding to alerts in the Lilith
platform.
Quick Reference#
| Category |
Document |
Common Alerts |
| SLO Alerts |
Service Level Objective violations |
SLOHighBurnRate, SLOErrorBudgetLow |
| Infrastructure Alerts |
Kubernetes and node issues |
KubePodCrashLooping, KubeNodeNotReady |
| Application Alerts |
Service-specific issues |
HighErrorRate, ServiceDown |
Alert Response Workflow#
- Acknowledge - Acknowledge the alert in PagerDuty/Slack
- Assess - Determine severity and impact
- Investigate - Follow runbook investigation steps
- Remediate - Apply appropriate fixes
- Communicate - Update stakeholders
- Document - Create incident report if needed
Severity Levels#
| Severity |
Response Time |
Who to Page |
| Critical |
Immediate |
On-call SRE |
| Warning |
30 minutes |
Service team |
| Info |
Next business day |
Service team |
Alert Dashboards#