# Alert Routing

Owner: platform team Last reviewed: 2026-05-28

How alerts move from the monitoring layer (Prometheus, CloudWatch, Grafana) to
the human (Slack, PagerDuty). This doc is the source of truth for routing — if a
service emits an alert and isn't in the table below, the on-call won't see it.

## Channels

| Channel                            | Purpose                                                | Quiet hours?                      |
| ---------------------------------- | ------------------------------------------------------ | --------------------------------- |
| `#alerts-platform` (Slack)         | All SEV-3 and FYI events                               | Yes (snooze noisy ones overnight) |
| `#alerts-prod` (Slack)             | SEV-1 / SEV-2 production events; broadcasts to on-call | No                                |
| `#alerts-staging` (Slack)          | Staging environment events                             | Yes                               |
| PagerDuty `oshun-platform` service | SEV-1 page + escalation                                | No — wakes on-call                |
| PagerDuty `oshun-domain-iris` etc. | Per-domain SEV-1 page; escalates to platform on no-ack | No                                |

## Routing table

| Source                                          | Filter      | Destination                         | Severity      |
| ----------------------------------------------- | ----------- | ----------------------------------- | ------------- |
| Prometheus `up == 0` for any production target  | Always      | PagerDuty `oshun-platform`          | SEV-1         |
| Prometheus error-rate > 5% over 5 min           | Per domain  | PagerDuty `oshun-domain-<domain>`   | SEV-2         |
| Prometheus p99 latency > SLO target over 10 min | Per domain  | `#alerts-prod` + domain channel     | SEV-2         |
| Prometheus deploy-marker                        | Per domain  | `#deploys-<domain>`                 | informational |
| CloudWatch RDS cpu > 85% for 15 min             | Always      | PagerDuty `oshun-platform`          | SEV-2         |
| CloudWatch RDS disk-free < 20%                  | Always      | PagerDuty `oshun-platform`          | SEV-1         |
| CloudWatch ECS task-died restart-loop           | Per service | `#alerts-prod` + per-domain channel | SEV-2         |
| GitHub Actions main-deploy failure              | Always      | `#alerts-platform`                  | SEV-3         |
| Sentry new-issue-spike                          | Per domain  | Domain channel                      | SEV-3         |
| ALB 5xx rate > 1%                               | Per service | PagerDuty `oshun-domain-<domain>`   | SEV-2         |

## Domain → PagerDuty service map

| Domain              | PagerDuty service                  | Primary                  | Secondary |
| ------------------- | ---------------------------------- | ------------------------ | --------- |
| aphrodite/payment   | `oshun-domain-aphrodite-payment`   | aphrodite-payment team   | platform  |
| aphrodite/streaming | `oshun-domain-aphrodite-streaming` | aphrodite-streaming team | platform  |
| iris/api            | `oshun-domain-iris`                | iris team                | platform  |
| lilith/svc-auth     | `oshun-domain-lilith-auth`         | lilith-platform team     | platform  |
| lilith/svc-ai       | `oshun-domain-lilith-ai`           | lilith-ml team           | platform  |
| nyx/api             | `oshun-domain-nyx`                 | nyx team                 | platform  |
| psyche              | `oshun-domain-psyche`              | psyche team              | platform  |
| tara                | `oshun-domain-tara`                | tara team                | platform  |
| veritas             | `oshun-domain-veritas`             | veritas team             | platform  |
| yemaya              | `oshun-domain-yemaya`              | yemaya team              | platform  |
| isis                | `oshun-domain-isis`                | isis team                | platform  |
| bellona             | `oshun-domain-bellona`             | bellona team             | platform  |

## Adding a new alert

1. Identify the SLO it ladders into (see [slos.md](slos.md)). If no SLO covers
   it, file the SLO first.
2. Pick the destination from the routing table above. If none apply, add a row.
3. Set up auto-resolve so the alert closes when the condition clears.
4. Wire a "runbook" link in the alert payload pointing to the appropriate
   `docs/runbooks/*` file.

## Suppression rules

- During a known deploy or scheduled maintenance window, set a PagerDuty
  maintenance window on the affected service.
- Never permanently silence an alert — if the alert is noisy, fix the underlying
  signal or change the threshold.
- Snoozes longer than 24 hours need a JIRA ticket linked.