Use this playbook during incidents to quickly isolate and remediate issues across Lilith services.
1) Classify the Failure#
- Availability: 5xx spikes, timeouts, failing health checks.
- Degradation: Elevated p95 latency, partial feature failures, slow queues.
- Data Integrity: Migrations, data loss/corruption indicators.
- Security: Suspected breach, secrets exposure, auth anomalies.
2) Immediate Checks (5 minutes)#
- Verify health endpoints:bash
for s in bff auth content ai; do curl -sf https://$s.lilith.app/health || echo "$s unhealthy"; done - Inspect service logs for the timeframe:bash
kubectl logs deploy/bff-production -n production --since=10m | tail -n 200 - Check error ratios and latency:bash
./scripts/observability/query-slo.sh --window 10m - Confirm dependency health (DB/Redis/queues).
3) Contain#
- Flip feature flags for risky paths.
- Scale up replicas if saturation is visible:bash
kubectl scale deploy/bff-production -n production --replicas=+2 - If release-related, route traffic back to stable (see rollback runbook).
4) Diagnose by Layer#
- Network/Ingress: Cert expiry, DNS changes, rate limiters, WAF blocks.
- App: Recent deployments, uncaught exceptions, config/env drift.
- Data: Slow queries, lock contention, migration effects, cache stampedes.
- External: Third-party dependency status pages and timeouts.
5) Remediate#
- Apply hotfix or rollback using the deployment rollback runbook.
- Clear poison messages / replay queues if stuck consumers.
- Rebuild caches behind feature flags to avoid thundering herd.
- For data issues, switch to read-only mode and engage DB on-call.
6) Communicate#
- Post status in
#api-alertswith impact, scope, and ETA. - Update the incident doc with timeline and actions.
- If customer impact is confirmed, page comms owner for outbound updates.
7) Verify and Close#
- Confirm SLOs recovered (error rate, latency).
- Run smoke tests:bash
npm run test:integration -- --grep "smoke" - Capture PIR with root cause, detection gap, and regression tests added.
- File follow-up tickets for infra/app fixes and assign owners.