# Alerting Procedures

Comprehensive guide for alerting configuration, routing, and response procedures
in the Lilith platform.

## Table of Contents

- [Overview](#overview)
- [Alert Architecture](#alert-architecture)
- [Severity Levels](#severity-levels)
- [Notification Channels](#notification-channels)
- [Alert Routing](#alert-routing)
- [Alert Configuration](#alert-configuration)
- [Response Procedures](#response-procedures)
- [Runbook Integration](#runbook-integration)
- [Best Practices](#best-practices)

## Overview

The Lilith platform uses a multi-tier alerting system to ensure rapid detection
and response to incidents.

### Alerting Stack

| Component     | Technology              | Purpose                  |
| ------------- | ----------------------- | ------------------------ |
| Alert Rules   | Prometheus/Alertmanager | Define alert conditions  |
| Routing       | Alertmanager            | Route alerts to channels |
| Notifications | Slack, PagerDuty, Email | Deliver alerts           |
| Runbooks      | Grafana OnCall          | Response procedures      |
| Dashboards    | Grafana                 | Alert visualization      |

### Alert Flow

```
┌─────────────────────────────────────────────────────────────────────────┐
│                          Alert Pipeline                                  │
│                                                                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │ Prometheus  │───▶│ Alertmanager│───▶│  Receivers  │                 │
│  │   Rules     │    │  (Routing)  │    │             │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
│        │                   │                  │                          │
│        ▼                   ▼                  ▼                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │  Evaluate   │    │   Group     │    │   Slack     │                 │
│  │ Conditions  │    │   Dedupe    │    │  PagerDuty  │                 │
│  │             │    │   Route     │    │   Email     │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
└─────────────────────────────────────────────────────────────────────────┘
```

## Alert Architecture

### Prometheus Alert Rules

Alert rules are defined in ConfigMaps and loaded by Prometheus:

```yaml
# deploy/terraform/modules/observability/alerting/rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: lilith-alerts
  namespace: monitoring
spec:
  groups:
    - name: availability
      interval: 30s
      rules:
        - alert: ServiceDown
          expr: up{job=~"lilith-.*"} == 0
          for: 1m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: 'Service {{ $labels.job }} is down'
            description:
              'Service {{ $labels.job }} has been down for more than 1 minute.'
            runbook_url: 'https://docs.lilith.app/runbooks/service-down'
            dashboard_url: 'https://grafana.lilith.app/d/service-health'

        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.05
          for: 5m
          labels:
            severity: warning
            team: '{{ $labels.service }}'
          annotations:
            summary: 'High error rate on {{ $labels.service }}'
            description:
              'Error rate is {{ $value | humanizePercentage }} on {{
              $labels.service }}'
            runbook_url: 'https://docs.lilith.app/runbooks/high-error-rate'
```

### Alertmanager Configuration

```yaml
# deploy/terraform/modules/observability/alertmanager/config.yaml
global:
  resolve_timeout: 5m
  slack_api_url: '${SLACK_WEBHOOK_URL}'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Inhibition rules to suppress lower-severity alerts
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'service']

  - source_match:
      alertname: ServiceDown
    target_match_re:
      alertname: .+
    equal: ['service']

# Alert routing tree
route:
  receiver: default-receiver
  group_by: ['alertname', 'severity', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts - immediate PagerDuty
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h
      continue: true

    # Warning alerts - Slack
    - match:
        severity: warning
      receiver: slack-warnings
      group_wait: 1m
      repeat_interval: 4h

    # Info alerts - Slack (low priority channel)
    - match:
        severity: info
      receiver: slack-info
      group_wait: 5m
      repeat_interval: 24h

    # Team-specific routing
    - match:
        team: platform
      receiver: platform-team

    - match:
        team: ai
      receiver: ai-team

    - match:
        team: content
      receiver: content-team

receivers:
  - name: default-receiver
    slack_configs:
      - channel: '#alerts-default'
        send_resolved: true

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '${PAGERDUTY_SERVICE_KEY}'
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ template "pagerduty.default.firing" . }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'

  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-warnings'
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
        actions:
          - type: button
            text: 'Runbook'
            url: '{{ .CommonAnnotations.runbook_url }}'
          - type: button
            text: 'Dashboard'
            url: '{{ .CommonAnnotations.dashboard_url }}'
          - type: button
            text: 'Silence'
            url: '{{ template "slack.silence_url" . }}'

  - name: slack-info
    slack_configs:
      - channel: '#alerts-info'
        send_resolved: false

  - name: platform-team
    slack_configs:
      - channel: '#platform-alerts'
        send_resolved: true
    pagerduty_configs:
      - service_key: '${PAGERDUTY_PLATFORM_KEY}'

  - name: ai-team
    slack_configs:
      - channel: '#ai-alerts'
        send_resolved: true

  - name: content-team
    slack_configs:
      - channel: '#content-alerts'
        send_resolved: true
```

## Severity Levels

### Severity Definitions

| Severity     | Response Time     | Description                              | Examples                             |
| ------------ | ----------------- | ---------------------------------------- | ------------------------------------ |
| **Critical** | < 15 min          | Service down, data loss, security breach | Database unreachable, auth failure   |
| **Warning**  | < 1 hour          | Degraded performance, approaching limits | High latency, disk 80%               |
| **Info**     | Next business day | Informational, non-urgent                | Deployment complete, backup finished |

### Severity Matrix

```
┌────────────────────────────────────────────────────────────────────────┐
│                     Severity Classification Matrix                      │
├────────────────────┬───────────┬───────────┬───────────┬──────────────┤
│ Impact / Urgency   │   High    │  Medium   │    Low    │   Minimal    │
├────────────────────┼───────────┼───────────┼───────────┼──────────────┤
│ High               │ CRITICAL  │ CRITICAL  │  WARNING  │   WARNING    │
│ Medium             │ CRITICAL  │  WARNING  │  WARNING  │    INFO      │
│ Low                │  WARNING  │  WARNING  │   INFO    │    INFO      │
│ Minimal            │  WARNING  │   INFO    │   INFO    │   (none)     │
└────────────────────┴───────────┴───────────┴───────────┴──────────────┘
```

### Severity Labels in Alerts

```yaml
# Critical - Immediate action required
labels:
  severity: critical
  priority: P1
  escalation_minutes: 15

# Warning - Action required within 1 hour
labels:
  severity: warning
  priority: P2
  escalation_minutes: 60

# Info - For awareness
labels:
  severity: info
  priority: P3
```

## Notification Channels

### Slack Configuration

```yaml
# Slack notification template
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# templates/slack.tmpl
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt | date "2006-01-02 15:04:05 MST" }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
---
{{ end }}
{{ end }}
```

### PagerDuty Configuration

```yaml
pagerduty_configs:
  - service_key: '${PAGERDUTY_SERVICE_KEY}'
    severity: '{{ .Labels.severity }}'
    client: 'Lilith Alertmanager'
    client_url: 'https://alertmanager.lilith.app'
    description: '{{ .CommonAnnotations.summary }}'
    details:
      firing: '{{ template "pagerduty.default.firing" . }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.resolved" . }}'
    links:
      - href: '{{ .CommonAnnotations.runbook_url }}'
        text: Runbook
      - href: '{{ .CommonAnnotations.dashboard_url }}'
        text: Dashboard
    images:
      - src:
          'https://grafana.lilith.app/render/d-solo/service-{{ .Labels.service
          }}'
        alt: 'Service Dashboard'
```

### Email Configuration

```yaml
email_configs:
  - to: 'oncall@lilith.app'
    from: 'alerts@lilith.app'
    smarthost: 'smtp.ses.amazonaws.com:587'
    auth_username: '${SES_USERNAME}'
    auth_password: '${SES_PASSWORD}'
    send_resolved: true
    headers:
      Subject: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
    html: '{{ template "email.html" . }}'
```

## Alert Routing

### Route Configuration

```yaml
# Detailed routing example
route:
  # Default route
  receiver: default-receiver
  group_by: ['alertname', 'severity', 'service']

  routes:
    # Production critical - page immediately
    - match:
        severity: critical
        environment: production
      receiver: pagerduty-critical
      group_wait: 0s

    # Staging alerts - Slack only
    - match:
        environment: staging
      receiver: slack-staging

    # Database alerts - DBA team
    - match_re:
        alertname: (PostgreSQL|RDS|Database).*
      receiver: dba-team

    # AI service alerts - AI team
    - match:
        service: ai
      receiver: ai-team
      routes:
        - match:
            alertname: GPUOutOfMemory
          receiver: ai-team-gpu

    # Payment alerts - highest priority
    - match:
        service: payment
      receiver: payment-critical
      group_wait: 0s
```

### Silencing Alerts

```bash
# Create a silence via amtool
amtool silence add alertname=ServiceDown service=auth \
  --comment="Planned maintenance" \
  --author="ops@lilith.app" \
  --duration=2h

# List active silences
amtool silence query

# Expire a silence
amtool silence expire <silence-id>
```

### Inhibition Rules

```yaml
inhibit_rules:
  # If cluster is down, suppress service alerts
  - source_match:
      alertname: KubernetesClusterDown
    target_match_re:
      alertname: .+
    equal: ['cluster']

  # If node is down, suppress pod alerts on that node
  - source_match:
      alertname: KubernetesNodeDown
    target_match:
      alertname: KubernetesPodNotReady
    equal: ['node']

  # Suppress lower severity when higher exists
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'service']
```

## Alert Configuration

### Standard Alert Template

```yaml
- alert: AlertName
  expr: <PromQL expression>
  for: <pending duration>
  labels:
    severity: <critical|warning|info>
    team: <team-name>
    service: <service-name>
    environment: '{{ $externalLabels.environment }}'
  annotations:
    summary: 'Short description'
    description: 'Detailed description with {{ $value }}'
    runbook_url: 'https://docs.lilith.app/runbooks/<alert-name>'
    dashboard_url: 'https://grafana.lilith.app/d/<dashboard-id>'
```

### Common Alert Rules

#### Infrastructure Alerts

```yaml
groups:
  - name: infrastructure
    rules:
      - alert: NodeHighCPU
        expr:
          100 - (avg by(node) (rate(node_cpu_seconds_total{mode="idle"}[5m])) *
          100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Node {{ $labels.node }} CPU usage is high'
          description: 'CPU usage is {{ $value | printf "%.1f" }}%'

      - alert: NodeHighMemory
        expr:
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) *
          100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Node {{ $labels.node }} memory usage is high'

      - alert: DiskSpaceLow
        expr:
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
          80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Disk space low on {{ $labels.node }}'

      - alert: DiskSpaceCritical
        expr:
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
          95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Disk space critical on {{ $labels.node }}'
```

#### Application Alerts

```yaml
groups:
  - name: application
    rules:
      - alert: HighLatencyP99
        expr:
          histogram_quantile(0.99,
          rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High P99 latency on {{ $labels.service }}'
          description: 'P99 latency is {{ $value | printf "%.2f" }}s'

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High error rate on {{ $labels.service }}'

      - alert: ServiceDown
        expr: up{job=~"lilith-.*"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Service {{ $labels.job }} is down'

      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Pod {{ $labels.pod }} is crash looping'
```

#### Database Alerts

```yaml
groups:
  - name: database
    rules:
      - alert: PostgreSQLDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'PostgreSQL instance {{ $labels.instance }} is down'

      - alert: PostgreSQLHighConnections
        expr:
          sum(pg_stat_activity_count) by (instance) /
          pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'PostgreSQL connections at {{ $value | humanizePercentage }}'

      - alert: PostgreSQLSlowQueries
        expr:
          rate(pg_stat_statements_seconds_total[5m]) /
          rate(pg_stat_statements_calls_total[5m]) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Slow queries detected on {{ $labels.instance }}'

      - alert: RedisDown
        expr: redis_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Redis instance {{ $labels.instance }} is down'
```

## Response Procedures

### On-Call Response Flow

```
┌─────────────────────────────────────────────────────────────────────────┐
│                      Incident Response Flow                              │
│                                                                          │
│  ┌────────────┐    ┌────────────┐    ┌────────────┐    ┌────────────┐  │
│  │   Alert    │───▶│   Triage   │───▶│  Respond   │───▶│  Resolve   │  │
│  │  Received  │    │            │    │            │    │            │  │
│  └────────────┘    └────────────┘    └────────────┘    └────────────┘  │
│        │                 │                  │                 │          │
│        ▼                 ▼                  ▼                 ▼          │
│   Acknowledge       Assess impact      Follow runbook    Document       │
│   in PagerDuty      Check dashboards   Escalate if       Post-mortem    │
│                     Review logs        needed                            │
└─────────────────────────────────────────────────────────────────────────┘
```

### Response Checklist

```markdown
## Critical Alert Response Checklist

### Immediate (0-5 minutes)

- [ ] Acknowledge alert in PagerDuty/Slack
- [ ] Open service dashboard
- [ ] Check for related alerts
- [ ] Identify impacted users/services

### Triage (5-15 minutes)

- [ ] Review recent deployments
- [ ] Check logs for errors
- [ ] Identify root cause or symptoms
- [ ] Communicate status to stakeholders

### Response (15+ minutes)

- [ ] Follow runbook procedures
- [ ] Apply fix or mitigation
- [ ] Verify resolution
- [ ] Clear alert

### Post-Incident

- [ ] Document timeline
- [ ] Create post-mortem if P1/P2
- [ ] Update runbooks if needed
```

### Escalation Matrix

| Level | Time    | Action                      |
| ----- | ------- | --------------------------- |
| L1    | 0 min   | Primary on-call paged       |
| L2    | 15 min  | Secondary on-call added     |
| L3    | 30 min  | Team lead notified          |
| L4    | 1 hour  | Engineering manager engaged |
| L5    | 2 hours | VP Engineering notified     |

## Runbook Integration

### Linking Alerts to Runbooks

```yaml
annotations:
  runbook_url:
    'https://docs.lilith.app/runbooks/{{ $labels.alertname | toLower }}'
```

### Runbook Template

```markdown
# [AlertName] Runbook

## Alert Details

- **Severity**: Critical/Warning/Info
- **Team**: Platform/AI/Content
- **Service**: service-name

## Description

What this alert means and why it fires.

## Impact

- User-facing impact
- Business impact
- Data impact

## Investigation Steps

1. Check the dashboard: [link]
2. Review logs: `kubectl logs -n production -l app=service-name`
3. Check recent deployments
4. Review metrics

## Resolution Steps

1. Step-by-step fix
2. Verification steps
3. Rollback procedure if needed

## Prevention

- How to prevent recurrence
- Recommended monitoring improvements
```

## Best Practices

### Alert Design

1. **Be specific**: Alert on symptoms, not causes
2. **Include context**: Add relevant labels and annotations
3. **Set appropriate thresholds**: Avoid alert fatigue
4. **Use pending duration**: Prevent flapping alerts
5. **Link runbooks**: Every alert should have a runbook

### Alert Hygiene

```yaml
# Good: Specific, actionable
- alert: AuthServiceHighLatencyP99
  expr:
    histogram_quantile(0.99,
    rate(http_request_duration_seconds_bucket{service="auth"}[5m])) > 0.5
  for: 5m
  annotations:
    summary: 'Auth service P99 latency exceeds 500ms'
    runbook_url: '...'

# Bad: Too generic, no context
- alert: HighLatency
  expr:
    histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
```

### Reducing Alert Fatigue

1. **Group related alerts**: Use `group_by` effectively
2. **Use inhibition**: Suppress redundant alerts
3. **Set repeat intervals**: Don't spam the same alert
4. **Review regularly**: Remove or adjust noisy alerts

### Testing Alerts

```bash
# Test alert rule syntax
promtool check rules rules.yaml

# Test alertmanager config
amtool check-config alertmanager.yml

# Send test alert
curl -X POST http://alertmanager:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning"
    },
    "annotations": {
      "summary": "This is a test alert"
    }
  }]'
```

## Troubleshooting

### Common Issues

| Issue             | Cause                 | Solution               |
| ----------------- | --------------------- | ---------------------- |
| Alert not firing  | Expression wrong      | Test in Prometheus UI  |
| Alert not routing | Route mismatch        | Check route labels     |
| No notifications  | Receiver config wrong | Test receiver endpoint |
| Too many alerts   | Thresholds too low    | Adjust thresholds      |

### Debug Commands

```bash
# Check Alertmanager status
curl http://alertmanager:9093/api/v1/status

# View active alerts
curl http://alertmanager:9093/api/v1/alerts

# Check routing
amtool config routes show --config.file=alertmanager.yml

# Test routing for specific labels
amtool config routes test --config.file=alertmanager.yml \
  severity=critical service=auth
```

## Related Documentation

- [Logging Standards](../LOGGING_STANDARDS.md) - Log correlation
- [Tracing Standards](./tracing-standards.md) - Trace context
- [Metrics Dashboard](../runbooks/metrics-dashboard.md) - Dashboard reference
- [On-Call Procedures](../runbooks/on-call/) - On-call rotation
