Eve library

Observability Stack

Guide

Revision 1

On this page

Comprehensive telemetry is foundational for Lilith’s reliability and compliance posture. The platform standardises on a Kubernetes-native observability stack built around Prometheus for metrics, Loki for logs, Tempo for trace storage, Grafana for visualisation, and the OpenTelemetry Collector for signal fan-in. All configuration is codified in Terraform and validated automatically inside CI so that infrastructure, documentation, and runtime behaviour stay aligned.

Metrics Architecture#

  • Signal collection – Every service exposes /metrics with Prometheus-compliant counters, gauges, histograms, and summaries. Service discovery is handled through Kubernetes annotations and ServiceMonitor objects.
  • Scraping and retention – The kube-prometheus-stack Helm release collects control-plane, node, and application metrics. Default scrape interval is 30s, and retention defaults to 30 days with remote write hooks to long-term storage buckets.
  • Dashboards – Core Grafana boards ship with the Terraform module (lilith-overview, service-metrics, kubernetes-cluster). Business KPI dashboards are generated from the analytics service and published via the observability API.
  • SLO instrumentation – Error budgets and latency targets are captured as recording rules and surfaced in Grafana with burn-rate visualisations.

Logging Pipeline#

  • Collection – Promtail runs as a DaemonSet to tail application, ingress, and system logs. Kubernetes labels are preserved to power namespace/service filtering.
  • Aggregation – Loki receives structured logs with immutable object storage (S3/MinIO) and BoltDB-shipper indexes. Tenant separation is achieved with per-service label isolation.
  • Retention – Application logs default to 30 days, audit/security logs default to 180 days with WORM buckets. Curated retention classes are documented in the privacy and compliance controls.
  • Access – Read permissions are enforced through Grafana RBAC and service-specific API tokens issued by the observability service.

Distributed Tracing with Tempo#

  • Ingestion – The OpenTelemetry Collector sidecars export traces, metrics, and logs to the Tempo distributor. Both gRPC and HTTP ingest endpoints are enabled.
  • Storage – Tempo stores spans in S3-compatible object storage with block compaction (compactor component) and a query frontend optimised for high-cardinality spans.
  • Query – Jaeger and Grafana Tempo datasources are enabled so that existing workflows can leverage trace search, service dependency graphs, and exemplar overlays on metrics dashboards.
  • Retention – Default trace retention is 14 days with per-service overrides for regulated workloads. Compaction guarantees cost-effective storage while preserving span metadata.

Alerting and Incident Response#

  • Alerting – Alertmanager routes SLO, infrastructure, and security alerts to PagerDuty, Slack, and the incident management bridge. Routing rules are version-controlled inside Terraform and validated by the observability check script.
  • Runbooks – Critical dashboards embed runbook links and severity-specific mitigation steps. The incident response handbook references this document for escalation contacts and verification flows.
  • Noise reduction – Multi-window, multi-burn-rate alerts ensure actionable paging. Warning notifications post to observability Slack channels for triage without paging the on-call responder.

Data Retention and Privacy Controls#

  • PII scrubbing – Ingestion pipelines scrub sensitive values before persistence. Structured logs apply field-level redaction policies driven by the privacy registry.
  • Lifecycle policies – Metrics, logs, and traces all carry TTLs defined by Terraform variables. Tempo’s retention, Loki’s compaction, and Prometheus remote write policies are parameterised per environment.
  • Auditability – All configuration changes are versioned via GitOps. The validation script asserts that alert rules, dashboards, and retention values remain in sync with policy definitions.

Deployment Automation#

  • Terraform module modules/observability provisions Prometheus, Grafana, Alertmanager, Loki, Promtail, Tempo, and Jaeger.
  • CI executes npm run observability:validate to check dashboards, alert rules, and Tempo storage wiring on every pull request.
  • Argo CD reconciles Helm releases continuously; drift detection on monitoring namespaces is enforced through reconciler alerts.
  • Runbooks in docs/runbooks/procedures/monitoring.md describe operational hand-offs, while this document captures the canonical topology and configuration guardrails.