Disciplines · Operations

Alert Routing

1.

5sections2 minread
Markdown source
On this page

Owner: platform team Last reviewed: 2026-05-28

How alerts move from the monitoring layer (Prometheus, CloudWatch, Grafana) to the human (Slack, PagerDuty). This doc is the source of truth for routing — if a service emits an alert and isn't in the table below, the on-call won't see it.

Channels#

Channel Purpose Quiet hours?
#alerts-platform (Slack) All SEV-3 and FYI events Yes (snooze noisy ones overnight)
#alerts-prod (Slack) SEV-1 / SEV-2 production events; broadcasts to on-call No
#alerts-staging (Slack) Staging environment events Yes
PagerDuty oshun-platform service SEV-1 page + escalation No — wakes on-call
PagerDuty oshun-domain-iris etc. Per-domain SEV-1 page; escalates to platform on no-ack No

Routing table#

Source Filter Destination Severity
Prometheus up == 0 for any production target Always PagerDuty oshun-platform SEV-1
Prometheus error-rate > 5% over 5 min Per domain PagerDuty oshun-domain-<domain> SEV-2
Prometheus p99 latency > SLO target over 10 min Per domain #alerts-prod + domain channel SEV-2
Prometheus deploy-marker Per domain #deploys-<domain> informational
CloudWatch RDS cpu > 85% for 15 min Always PagerDuty oshun-platform SEV-2
CloudWatch RDS disk-free < 20% Always PagerDuty oshun-platform SEV-1
CloudWatch ECS task-died restart-loop Per service #alerts-prod + per-domain channel SEV-2
GitHub Actions main-deploy failure Always #alerts-platform SEV-3
Sentry new-issue-spike Per domain Domain channel SEV-3
ALB 5xx rate > 1% Per service PagerDuty oshun-domain-<domain> SEV-2

Domain → PagerDuty service map#

Domain PagerDuty service Primary Secondary
aphrodite/payment oshun-domain-aphrodite-payment aphrodite-payment team platform
aphrodite/streaming oshun-domain-aphrodite-streaming aphrodite-streaming team platform
iris/api oshun-domain-iris iris team platform
lilith/svc-auth oshun-domain-lilith-auth lilith-platform team platform
lilith/svc-ai oshun-domain-lilith-ai lilith-ml team platform
nyx/api oshun-domain-nyx nyx team platform
psyche oshun-domain-psyche psyche team platform
tara oshun-domain-tara tara team platform
veritas oshun-domain-veritas veritas team platform
yemaya oshun-domain-yemaya yemaya team platform
isis oshun-domain-isis isis team platform
bellona oshun-domain-bellona bellona team platform

Adding a new alert#

  1. Identify the SLO it ladders into (see slos.md). If no SLO covers it, file the SLO first.
  2. Pick the destination from the routing table above. If none apply, add a row.
  3. Set up auto-resolve so the alert closes when the condition clears.
  4. Wire a "runbook" link in the alert payload pointing to the appropriate docs/runbooks/* file.

Suppression rules#

  • During a known deploy or scheduled maintenance window, set a PagerDuty maintenance window on the affected service.
  • Never permanently silence an alert — if the alert is noisy, fix the underlying signal or change the threshold.
  • Snoozes longer than 24 hours need a JIRA ticket linked.