Comprehensive telemetry is foundational for Lilith’s reliability and compliance posture. The platform standardises on a Kubernetes-native observability stack built around Prometheus for metrics, Loki for logs, Tempo for trace storage, Grafana for visualisation, and the OpenTelemetry Collector for signal fan-in. All configuration is codified in Terraform and validated automatically inside CI so that infrastructure, documentation, and runtime behaviour stay aligned.
Metrics Architecture#
- Signal collection – Every service exposes
/metricswith Prometheus-compliant counters, gauges, histograms, and summaries. Service discovery is handled through Kubernetes annotations andServiceMonitorobjects. - Scraping and retention – The
kube-prometheus-stackHelm release collects control-plane, node, and application metrics. Default scrape interval is 30s, and retention defaults to 30 days with remote write hooks to long-term storage buckets. - Dashboards – Core Grafana boards ship with the Terraform module
(
lilith-overview,service-metrics,kubernetes-cluster). Business KPI dashboards are generated from the analytics service and published via the observability API. - SLO instrumentation – Error budgets and latency targets are captured as recording rules and surfaced in Grafana with burn-rate visualisations.
Logging Pipeline#
- Collection – Promtail runs as a DaemonSet to tail application, ingress, and system logs. Kubernetes labels are preserved to power namespace/service filtering.
- Aggregation – Loki receives structured logs with immutable object storage (S3/MinIO) and BoltDB-shipper indexes. Tenant separation is achieved with per-service label isolation.
- Retention – Application logs default to 30 days, audit/security logs default to 180 days with WORM buckets. Curated retention classes are documented in the privacy and compliance controls.
- Access – Read permissions are enforced through Grafana RBAC and service-specific API tokens issued by the observability service.
Distributed Tracing with Tempo#
- Ingestion – The OpenTelemetry Collector sidecars export traces, metrics, and logs to the Tempo distributor. Both gRPC and HTTP ingest endpoints are enabled.
- Storage – Tempo stores spans in S3-compatible object storage with block
compaction (
compactorcomponent) and a query frontend optimised for high-cardinality spans. - Query – Jaeger and Grafana Tempo datasources are enabled so that existing workflows can leverage trace search, service dependency graphs, and exemplar overlays on metrics dashboards.
- Retention – Default trace retention is 14 days with per-service overrides for regulated workloads. Compaction guarantees cost-effective storage while preserving span metadata.
Alerting and Incident Response#
- Alerting – Alertmanager routes SLO, infrastructure, and security alerts to PagerDuty, Slack, and the incident management bridge. Routing rules are version-controlled inside Terraform and validated by the observability check script.
- Runbooks – Critical dashboards embed runbook links and severity-specific mitigation steps. The incident response handbook references this document for escalation contacts and verification flows.
- Noise reduction – Multi-window, multi-burn-rate alerts ensure actionable paging. Warning notifications post to observability Slack channels for triage without paging the on-call responder.
Data Retention and Privacy Controls#
- PII scrubbing – Ingestion pipelines scrub sensitive values before persistence. Structured logs apply field-level redaction policies driven by the privacy registry.
- Lifecycle policies – Metrics, logs, and traces all carry TTLs defined by Terraform variables. Tempo’s retention, Loki’s compaction, and Prometheus remote write policies are parameterised per environment.
- Auditability – All configuration changes are versioned via GitOps. The validation script asserts that alert rules, dashboards, and retention values remain in sync with policy definitions.
Deployment Automation#
- Terraform module
modules/observabilityprovisions Prometheus, Grafana, Alertmanager, Loki, Promtail, Tempo, and Jaeger. - CI executes
npm run observability:validateto check dashboards, alert rules, and Tempo storage wiring on every pull request. - Argo CD reconciles Helm releases continuously; drift detection on monitoring namespaces is enforced through reconciler alerts.
- Runbooks in
docs/runbooks/procedures/monitoring.mddescribe operational hand-offs, while this document captures the canonical topology and configuration guardrails.