# Observability Stack

Comprehensive telemetry is foundational for Lilith’s reliability and compliance
posture. The platform standardises on a Kubernetes-native observability stack
built around Prometheus for metrics, Loki for logs, Tempo for trace storage,
Grafana for visualisation, and the OpenTelemetry Collector for signal fan-in.
All configuration is codified in Terraform and validated automatically inside CI
so that infrastructure, documentation, and runtime behaviour stay aligned.

## Metrics Architecture

- **Signal collection** – Every service exposes `/metrics` with
  Prometheus-compliant counters, gauges, histograms, and summaries. Service
  discovery is handled through Kubernetes annotations and `ServiceMonitor`
  objects.
- **Scraping and retention** – The `kube-prometheus-stack` Helm release collects
  control-plane, node, and application metrics. Default scrape interval is 30s,
  and retention defaults to 30 days with remote write hooks to long-term storage
  buckets.
- **Dashboards** – Core Grafana boards ship with the Terraform module
  (`lilith-overview`, `service-metrics`, `kubernetes-cluster`). Business KPI
  dashboards are generated from the analytics service and published via the
  observability API.
- **SLO instrumentation** – Error budgets and latency targets are captured as
  recording rules and surfaced in Grafana with burn-rate visualisations.

## Logging Pipeline

- **Collection** – Promtail runs as a DaemonSet to tail application, ingress,
  and system logs. Kubernetes labels are preserved to power namespace/service
  filtering.
- **Aggregation** – Loki receives structured logs with immutable object storage
  (S3/MinIO) and BoltDB-shipper indexes. Tenant separation is achieved with
  per-service label isolation.
- **Retention** – Application logs default to 30 days, audit/security logs
  default to 180 days with WORM buckets. Curated retention classes are
  documented in the privacy and compliance controls.
- **Access** – Read permissions are enforced through Grafana RBAC and
  service-specific API tokens issued by the observability service.

## Distributed Tracing with Tempo

- **Ingestion** – The OpenTelemetry Collector sidecars export traces, metrics,
  and logs to the Tempo distributor. Both gRPC and HTTP ingest endpoints are
  enabled.
- **Storage** – Tempo stores spans in S3-compatible object storage with block
  compaction (`compactor` component) and a query frontend optimised for
  high-cardinality spans.
- **Query** – Jaeger and Grafana Tempo datasources are enabled so that existing
  workflows can leverage trace search, service dependency graphs, and exemplar
  overlays on metrics dashboards.
- **Retention** – Default trace retention is 14 days with per-service overrides
  for regulated workloads. Compaction guarantees cost-effective storage while
  preserving span metadata.

## Alerting and Incident Response

- **Alerting** – Alertmanager routes SLO, infrastructure, and security alerts to
  PagerDuty, Slack, and the incident management bridge. Routing rules are
  version-controlled inside Terraform and validated by the observability check
  script.
- **Runbooks** – Critical dashboards embed runbook links and severity-specific
  mitigation steps. The incident response handbook references this document for
  escalation contacts and verification flows.
- **Noise reduction** – Multi-window, multi-burn-rate alerts ensure actionable
  paging. Warning notifications post to observability Slack channels for triage
  without paging the on-call responder.

## Data Retention and Privacy Controls

- **PII scrubbing** – Ingestion pipelines scrub sensitive values before
  persistence. Structured logs apply field-level redaction policies driven by
  the privacy registry.
- **Lifecycle policies** – Metrics, logs, and traces all carry TTLs defined by
  Terraform variables. Tempo’s retention, Loki’s compaction, and Prometheus
  remote write policies are parameterised per environment.
- **Auditability** – All configuration changes are versioned via GitOps. The
  validation script asserts that alert rules, dashboards, and retention values
  remain in sync with policy definitions.

## Deployment Automation

- Terraform module `modules/observability` provisions Prometheus, Grafana,
  Alertmanager, Loki, Promtail, Tempo, and Jaeger.
- CI executes `npm run observability:validate` to check dashboards, alert rules,
  and Tempo storage wiring on every pull request.
- Argo CD reconciles Helm releases continuously; drift detection on monitoring
  namespaces is enforced through reconciler alerts.
- Runbooks in `docs/runbooks/procedures/monitoring.md` describe operational
  hand-offs, while this document captures the canonical topology and
  configuration guardrails.
