Eve library

Alert Runbooks

Runbook

Revision 1

On this page

This directory contains runbooks for responding to alerts in the Lilith platform.

Quick Reference#

Category Document Common Alerts
SLO Alerts Service Level Objective violations SLOHighBurnRate, SLOErrorBudgetLow
Infrastructure Alerts Kubernetes and node issues KubePodCrashLooping, KubeNodeNotReady
Application Alerts Service-specific issues HighErrorRate, ServiceDown

Alert Response Workflow#

  1. Acknowledge - Acknowledge the alert in PagerDuty/Slack
  2. Assess - Determine severity and impact
  3. Investigate - Follow runbook investigation steps
  4. Remediate - Apply appropriate fixes
  5. Communicate - Update stakeholders
  6. Document - Create incident report if needed

Severity Levels#

Severity Response Time Who to Page
Critical Immediate On-call SRE
Warning 30 minutes Service team
Info Next business day Service team

Alert Dashboards#

On-Call Contacts#