Comprehensive guide for alerting configuration, routing, and response procedures in the Lilith platform.
Table of Contents#
- Overview
- Alert Architecture
- Severity Levels
- Notification Channels
- Alert Routing
- Alert Configuration
- Response Procedures
- Runbook Integration
- Best Practices
Overview#
The Lilith platform uses a multi-tier alerting system to ensure rapid detection and response to incidents.
Alerting Stack#
| Component | Technology | Purpose |
|---|---|---|
| Alert Rules | Prometheus/Alertmanager | Define alert conditions |
| Routing | Alertmanager | Route alerts to channels |
| Notifications | Slack, PagerDuty, Email | Deliver alerts |
| Runbooks | Grafana OnCall | Response procedures |
| Dashboards | Grafana | Alert visualization |
Alert Flow#
text
┌─────────────────────────────────────────────────────────────────────────┐
│ Alert Pipeline │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Prometheus │───▶│ Alertmanager│───▶│ Receivers │ │
│ │ Rules │ │ (Routing) │ │ │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Evaluate │ │ Group │ │ Slack │ │
│ │ Conditions │ │ Dedupe │ │ PagerDuty │ │
│ │ │ │ Route │ │ Email │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Alert Architecture#
Prometheus Alert Rules#
Alert rules are defined in ConfigMaps and loaded by Prometheus:
yaml
# deploy/terraform/modules/observability/alerting/rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: lilith-alerts
namespace: monitoring
spec:
groups:
- name: availability
interval: 30s
rules:
- alert: ServiceDown
expr: up{job=~"lilith-.*"} == 0
for: 1m
labels:
severity: critical
team: platform
annotations:
summary: 'Service {{ $labels.job }} is down'
description:
'Service {{ $labels.job }} has been down for more than 1 minute.'
runbook_url: 'https://docs.lilith.app/runbooks/service-down'
dashboard_url: 'https://grafana.lilith.app/d/service-health'
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
> 0.05
for: 5m
labels:
severity: warning
team: '{{ $labels.service }}'
annotations:
summary: 'High error rate on {{ $labels.service }}'
description:
'Error rate is {{ $value | humanizePercentage }} on {{
$labels.service }}'
runbook_url: 'https://docs.lilith.app/runbooks/high-error-rate'
Alertmanager Configuration#
yaml
# deploy/terraform/modules/observability/alertmanager/config.yaml
global:
resolve_timeout: 5m
slack_api_url: '${SLACK_WEBHOOK_URL}'
pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
# Inhibition rules to suppress lower-severity alerts
inhibit_rules:
- source_match:
severity: critical
target_match:
severity: warning
equal: ['alertname', 'service']
- source_match:
alertname: ServiceDown
target_match_re:
alertname: .+
equal: ['service']
# Alert routing tree
route:
receiver: default-receiver
group_by: ['alertname', 'severity', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# Critical alerts - immediate PagerDuty
- match:
severity: critical
receiver: pagerduty-critical
group_wait: 10s
repeat_interval: 1h
continue: true
# Warning alerts - Slack
- match:
severity: warning
receiver: slack-warnings
group_wait: 1m
repeat_interval: 4h
# Info alerts - Slack (low priority channel)
- match:
severity: info
receiver: slack-info
group_wait: 5m
repeat_interval: 24h
# Team-specific routing
- match:
team: platform
receiver: platform-team
- match:
team: ai
receiver: ai-team
- match:
team: content
receiver: content-team
receivers:
- name: default-receiver
slack_configs:
- channel: '#alerts-default'
send_resolved: true
- name: pagerduty-critical
pagerduty_configs:
- service_key: '${PAGERDUTY_SERVICE_KEY}'
severity: critical
description: '{{ .CommonAnnotations.summary }}'
details:
firing: '{{ template "pagerduty.default.firing" . }}'
runbook: '{{ .CommonAnnotations.runbook_url }}'
- name: slack-warnings
slack_configs:
- channel: '#alerts-warnings'
send_resolved: true
title: '{{ template "slack.title" . }}'
text: '{{ template "slack.text" . }}'
actions:
- type: button
text: 'Runbook'
url: '{{ .CommonAnnotations.runbook_url }}'
- type: button
text: 'Dashboard'
url: '{{ .CommonAnnotations.dashboard_url }}'
- type: button
text: 'Silence'
url: '{{ template "slack.silence_url" . }}'
- name: slack-info
slack_configs:
- channel: '#alerts-info'
send_resolved: false
- name: platform-team
slack_configs:
- channel: '#platform-alerts'
send_resolved: true
pagerduty_configs:
- service_key: '${PAGERDUTY_PLATFORM_KEY}'
- name: ai-team
slack_configs:
- channel: '#ai-alerts'
send_resolved: true
- name: content-team
slack_configs:
- channel: '#content-alerts'
send_resolved: true
Severity Levels#
Severity Definitions#
| Severity | Response Time | Description | Examples |
|---|---|---|---|
| Critical | < 15 min | Service down, data loss, security breach | Database unreachable, auth failure |
| Warning | < 1 hour | Degraded performance, approaching limits | High latency, disk 80% |
| Info | Next business day | Informational, non-urgent | Deployment complete, backup finished |
Severity Matrix#
text
┌────────────────────────────────────────────────────────────────────────┐
│ Severity Classification Matrix │
├────────────────────┬───────────┬───────────┬───────────┬──────────────┤
│ Impact / Urgency │ High │ Medium │ Low │ Minimal │
├────────────────────┼───────────┼───────────┼───────────┼──────────────┤
│ High │ CRITICAL │ CRITICAL │ WARNING │ WARNING │
│ Medium │ CRITICAL │ WARNING │ WARNING │ INFO │
│ Low │ WARNING │ WARNING │ INFO │ INFO │
│ Minimal │ WARNING │ INFO │ INFO │ (none) │
└────────────────────┴───────────┴───────────┴───────────┴──────────────┘
Severity Labels in Alerts#
yaml
# Critical - Immediate action required
labels:
severity: critical
priority: P1
escalation_minutes: 15
# Warning - Action required within 1 hour
labels:
severity: warning
priority: P2
escalation_minutes: 60
# Info - For awareness
labels:
severity: info
priority: P3
Notification Channels#
Slack Configuration#
yaml
# Slack notification template
templates:
- '/etc/alertmanager/templates/*.tmpl'
# templates/slack.tmpl
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}
{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt | date "2006-01-02 15:04:05 MST" }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
---
{{ end }}
{{ end }}
PagerDuty Configuration#
yaml
pagerduty_configs:
- service_key: '${PAGERDUTY_SERVICE_KEY}'
severity: '{{ .Labels.severity }}'
client: 'Lilith Alertmanager'
client_url: 'https://alertmanager.lilith.app'
description: '{{ .CommonAnnotations.summary }}'
details:
firing: '{{ template "pagerduty.default.firing" . }}'
num_firing: '{{ .Alerts.Firing | len }}'
num_resolved: '{{ .Alerts.Resolved | len }}'
resolved: '{{ template "pagerduty.default.resolved" . }}'
links:
- href: '{{ .CommonAnnotations.runbook_url }}'
text: Runbook
- href: '{{ .CommonAnnotations.dashboard_url }}'
text: Dashboard
images:
- src:
'https://grafana.lilith.app/render/d-solo/service-{{ .Labels.service
}}'
alt: 'Service Dashboard'
Email Configuration#
yaml
email_configs:
- to: 'oncall@lilith.app'
from: 'alerts@lilith.app'
smarthost: 'smtp.ses.amazonaws.com:587'
auth_username: '${SES_USERNAME}'
auth_password: '${SES_PASSWORD}'
send_resolved: true
headers:
Subject: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
html: '{{ template "email.html" . }}'
Alert Routing#
Route Configuration#
yaml
# Detailed routing example
route:
# Default route
receiver: default-receiver
group_by: ['alertname', 'severity', 'service']
routes:
# Production critical - page immediately
- match:
severity: critical
environment: production
receiver: pagerduty-critical
group_wait: 0s
# Staging alerts - Slack only
- match:
environment: staging
receiver: slack-staging
# Database alerts - DBA team
- match_re:
alertname: (PostgreSQL|RDS|Database).*
receiver: dba-team
# AI service alerts - AI team
- match:
service: ai
receiver: ai-team
routes:
- match:
alertname: GPUOutOfMemory
receiver: ai-team-gpu
# Payment alerts - highest priority
- match:
service: payment
receiver: payment-critical
group_wait: 0s
Silencing Alerts#
bash
# Create a silence via amtool
amtool silence add alertname=ServiceDown service=auth \
--comment="Planned maintenance" \
--author="ops@lilith.app" \
--duration=2h
# List active silences
amtool silence query
# Expire a silence
amtool silence expire <silence-id>
Inhibition Rules#
yaml
inhibit_rules:
# If cluster is down, suppress service alerts
- source_match:
alertname: KubernetesClusterDown
target_match_re:
alertname: .+
equal: ['cluster']
# If node is down, suppress pod alerts on that node
- source_match:
alertname: KubernetesNodeDown
target_match:
alertname: KubernetesPodNotReady
equal: ['node']
# Suppress lower severity when higher exists
- source_match:
severity: critical
target_match:
severity: warning
equal: ['alertname', 'service']
Alert Configuration#
Standard Alert Template#
yaml
- alert: AlertName
expr: <PromQL expression>
for: <pending duration>
labels:
severity: <critical|warning|info>
team: <team-name>
service: <service-name>
environment: '{{ $externalLabels.environment }}'
annotations:
summary: 'Short description'
description: 'Detailed description with {{ $value }}'
runbook_url: 'https://docs.lilith.app/runbooks/<alert-name>'
dashboard_url: 'https://grafana.lilith.app/d/<dashboard-id>'
Common Alert Rules#
Infrastructure Alerts#
yaml
groups:
- name: infrastructure
rules:
- alert: NodeHighCPU
expr:
100 - (avg by(node) (rate(node_cpu_seconds_total{mode="idle"}[5m])) *
100) > 80
for: 10m
labels:
severity: warning
annotations:
summary: 'Node {{ $labels.node }} CPU usage is high'
description: 'CPU usage is {{ $value | printf "%.1f" }}%'
- alert: NodeHighMemory
expr:
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) *
100 > 85
for: 10m
labels:
severity: warning
annotations:
summary: 'Node {{ $labels.node }} memory usage is high'
- alert: DiskSpaceLow
expr:
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
80
for: 10m
labels:
severity: warning
annotations:
summary: 'Disk space low on {{ $labels.node }}'
- alert: DiskSpaceCritical
expr:
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
95
for: 5m
labels:
severity: critical
annotations:
summary: 'Disk space critical on {{ $labels.node }}'
Application Alerts#
yaml
groups:
- name: application
rules:
- alert: HighLatencyP99
expr:
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])) > 1
for: 5m
labels:
severity: warning
annotations:
summary: 'High P99 latency on {{ $labels.service }}'
description: 'P99 latency is {{ $value | printf "%.2f" }}s'
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: 'High error rate on {{ $labels.service }}'
- alert: ServiceDown
expr: up{job=~"lilith-.*"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: 'Service {{ $labels.job }} is down'
- alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 0
for: 5m
labels:
severity: warning
annotations:
summary: 'Pod {{ $labels.pod }} is crash looping'
Database Alerts#
yaml
groups:
- name: database
rules:
- alert: PostgreSQLDown
expr: pg_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: 'PostgreSQL instance {{ $labels.instance }} is down'
- alert: PostgreSQLHighConnections
expr:
sum(pg_stat_activity_count) by (instance) /
pg_settings_max_connections > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: 'PostgreSQL connections at {{ $value | humanizePercentage }}'
- alert: PostgreSQLSlowQueries
expr:
rate(pg_stat_statements_seconds_total[5m]) /
rate(pg_stat_statements_calls_total[5m]) > 1
for: 5m
labels:
severity: warning
annotations:
summary: 'Slow queries detected on {{ $labels.instance }}'
- alert: RedisDown
expr: redis_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: 'Redis instance {{ $labels.instance }} is down'
Response Procedures#
On-Call Response Flow#
text
┌─────────────────────────────────────────────────────────────────────────┐
│ Incident Response Flow │
│ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ Alert │───▶│ Triage │───▶│ Respond │───▶│ Resolve │ │
│ │ Received │ │ │ │ │ │ │ │
│ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ Acknowledge Assess impact Follow runbook Document │
│ in PagerDuty Check dashboards Escalate if Post-mortem │
│ Review logs needed │
└─────────────────────────────────────────────────────────────────────────┘
Response Checklist#
markdown
## Critical Alert Response Checklist
### Immediate (0-5 minutes)
- [ ] Acknowledge alert in PagerDuty/Slack
- [ ] Open service dashboard
- [ ] Check for related alerts
- [ ] Identify impacted users/services
### Triage (5-15 minutes)
- [ ] Review recent deployments
- [ ] Check logs for errors
- [ ] Identify root cause or symptoms
- [ ] Communicate status to stakeholders
### Response (15+ minutes)
- [ ] Follow runbook procedures
- [ ] Apply fix or mitigation
- [ ] Verify resolution
- [ ] Clear alert
### Post-Incident
- [ ] Document timeline
- [ ] Create post-mortem if P1/P2
- [ ] Update runbooks if needed
Escalation Matrix#
| Level | Time | Action |
|---|---|---|
| L1 | 0 min | Primary on-call paged |
| L2 | 15 min | Secondary on-call added |
| L3 | 30 min | Team lead notified |
| L4 | 1 hour | Engineering manager engaged |
| L5 | 2 hours | VP Engineering notified |
Runbook Integration#
Linking Alerts to Runbooks#
yaml
annotations:
runbook_url:
'https://docs.lilith.app/runbooks/{{ $labels.alertname | toLower }}'
Runbook Template#
markdown
# [AlertName] Runbook
## Alert Details
- **Severity**: Critical/Warning/Info
- **Team**: Platform/AI/Content
- **Service**: service-name
## Description
What this alert means and why it fires.
## Impact
- User-facing impact
- Business impact
- Data impact
## Investigation Steps
1. Check the dashboard: [link]
2. Review logs: `kubectl logs -n production -l app=service-name`
3. Check recent deployments
4. Review metrics
## Resolution Steps
1. Step-by-step fix
2. Verification steps
3. Rollback procedure if needed
## Prevention
- How to prevent recurrence
- Recommended monitoring improvements
Best Practices#
Alert Design#
- Be specific: Alert on symptoms, not causes
- Include context: Add relevant labels and annotations
- Set appropriate thresholds: Avoid alert fatigue
- Use pending duration: Prevent flapping alerts
- Link runbooks: Every alert should have a runbook
Alert Hygiene#
yaml
# Good: Specific, actionable
- alert: AuthServiceHighLatencyP99
expr:
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket{service="auth"}[5m])) > 0.5
for: 5m
annotations:
summary: 'Auth service P99 latency exceeds 500ms'
runbook_url: '...'
# Bad: Too generic, no context
- alert: HighLatency
expr:
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
Reducing Alert Fatigue#
- Group related alerts: Use
group_byeffectively - Use inhibition: Suppress redundant alerts
- Set repeat intervals: Don't spam the same alert
- Review regularly: Remove or adjust noisy alerts
Testing Alerts#
bash
# Test alert rule syntax
promtool check rules rules.yaml
# Test alertmanager config
amtool check-config alertmanager.yml
# Send test alert
curl -X POST http://alertmanager:9093/api/v1/alerts \
-H "Content-Type: application/json" \
-d '[{
"labels": {
"alertname": "TestAlert",
"severity": "warning"
},
"annotations": {
"summary": "This is a test alert"
}
}]'
Troubleshooting#
Common Issues#
| Issue | Cause | Solution |
|---|---|---|
| Alert not firing | Expression wrong | Test in Prometheus UI |
| Alert not routing | Route mismatch | Check route labels |
| No notifications | Receiver config wrong | Test receiver endpoint |
| Too many alerts | Thresholds too low | Adjust thresholds |
Debug Commands#
bash
# Check Alertmanager status
curl http://alertmanager:9093/api/v1/status
# View active alerts
curl http://alertmanager:9093/api/v1/alerts
# Check routing
amtool config routes show --config.file=alertmanager.yml
# Test routing for specific labels
amtool config routes test --config.file=alertmanager.yml \
severity=critical service=auth
Related Documentation#
- Logging Standards - Log correlation
- Tracing Standards - Trace context
- Metrics Dashboard - Dashboard reference
- On-Call Procedures - On-call rotation