# Troubleshooting Guide (Rapid Triage)

Use this playbook during incidents to quickly isolate and remediate issues
across Lilith services.

## 1) Classify the Failure

- **Availability:** 5xx spikes, timeouts, failing health checks.
- **Degradation:** Elevated p95 latency, partial feature failures, slow queues.
- **Data Integrity:** Migrations, data loss/corruption indicators.
- **Security:** Suspected breach, secrets exposure, auth anomalies.

## 2) Immediate Checks (5 minutes)

- Verify health endpoints:
  ```bash
  for s in bff auth content ai; do curl -sf https://$s.lilith.app/health || echo "$s unhealthy"; done
  ```
- Inspect service logs for the timeframe:
  ```bash
  kubectl logs deploy/bff-production -n production --since=10m | tail -n 200
  ```
- Check error ratios and latency:
  ```bash
  ./scripts/observability/query-slo.sh --window 10m
  ```
- Confirm dependency health (DB/Redis/queues).

## 3) Contain

- Flip feature flags for risky paths.
- Scale up replicas if saturation is visible:
  ```bash
  kubectl scale deploy/bff-production -n production --replicas=+2
  ```
- If release-related, route traffic back to stable (see rollback runbook).

## 4) Diagnose by Layer

- **Network/Ingress:** Cert expiry, DNS changes, rate limiters, WAF blocks.
- **App:** Recent deployments, uncaught exceptions, config/env drift.
- **Data:** Slow queries, lock contention, migration effects, cache stampedes.
- **External:** Third-party dependency status pages and timeouts.

## 5) Remediate

- Apply hotfix or rollback using the deployment rollback runbook.
- Clear poison messages / replay queues if stuck consumers.
- Rebuild caches behind feature flags to avoid thundering herd.
- For data issues, switch to read-only mode and engage DB on-call.

## 6) Communicate

- Post status in `#api-alerts` with impact, scope, and ETA.
- Update the incident doc with timeline and actions.
- If customer impact is confirmed, page comms owner for outbound updates.

## 7) Verify and Close

- Confirm SLOs recovered (error rate, latency).
- Run smoke tests:
  ```bash
  npm run test:integration -- --grep "smoke"
  ```
- Capture PIR with root cause, detection gap, and regression tests added.
- File follow-up tickets for infra/app fixes and assign owners.
