# Centralized Config and Secret Management

Lilith standardizes on HashiCorp Vault as the control plane for runtime configuration and sensitive material, with External Secrets Operator (ESO) bridging secrets into Kubernetes namespaces and the ConfigManager library enforcing schema validation inside each service. This document captures architecture decisions, operational procedures, and guardrails that keep credentials safe, rotated, and observable.

## Vault Architecture and Hardening

- **Deployment topology** – Vault runs as a highly available cluster (Raft storage + AWS KMS auto-unseal) in namespace `vault`. Helm values enforce TLS everywhere, dedicated storage classes, and separate I/O classes for audit and data volumes.
- **Network policy** – Only in-cluster callers in the same namespace may reach Vault’s HTTPS interface. Egress is restricted to DNS + AWS APIs for KMS interactions.
- **Authentication** – Kubernetes auth backend is enabled with dedicated roles per service account (`bff`, `auth`, `ai`, etc.) plus a shared `external-secrets` role for ESO. Tokens are scoped to the minimum TTL required (1 hour) and are renewable via the Vault client.
- **Secret engines** – KV v2 is mounted at `secret/` for configuration payloads, and the database engine under `database/` issues short-lived credentials for Postgres, Milvus, and analytics replicas. Audit logging is enabled with immutability enforced through remote storage.
- **Policies** – Policy files under `deploy/terraform/modules/vault/policies` describe explicit capability lists. ConfigMap-style reads live in read-only paths, while dynamic credentials are exposed via narrow prefixes so leaking one policy cannot escalate to another service.

## External Secrets Operator Integration

- **Operator** – Terraform installs the ESO Helm chart in namespace `external-secrets`, enabling CRDs, RBAC, and a service account bound to the `external-secrets` Vault role.
- **ClusterSecretStore** – A cluster-wide store points at `https://vault.vault.svc.cluster.local:8200`, uses Kubernetes auth, and reuses the Vault CA bundle from the `vault-tls` secret.
- **Sync workflow** – `ExternalSecret` resources declare which Vault paths should materialize as native Kubernetes `Secret` objects. A default `app-config` mapping fans out base configuration (database URLs, feature flags) into the application namespace while leaving ownership inside Vault.
- **Rotation cadence** – ESO reconciles every 60 seconds by default. When `SecretRotationManager` rotates material inside Vault, the next reconcile updates dependent secrets automatically without pods restarting.

## Configuration Distribution & Schema Enforcement

- **ConfigManager** – Services consume `services/lib/config-manager.js`, which loads hierarchical config (base → environment → service overrides) and validates against JSON Schema. Vault values take precedence over filesystem defaults.
- **External Credential Manager** – Provider keys (OpenAI, Anthropic, AWS) live under `secret/providers/*`. ConfigManager dynamically injects them into the service container, while masking occurs before logs or metrics ingestion.
- **Environment parity** – The Terraform stack publishes `ClusterSecretStore`, `ExternalSecret`, and namespace wiring per environment. CI verifies manifests so staging/prod drift cannot occur.

## Rotation, Auditing, and Break-glass Procedures

- **Rotation automation** – `SecretRotationManager` registers rotation hooks per secret. Database credentials rotate every 24h, API keys every 7d, and TLS certificates every 30d. ESO propagates fresh material to pods through `Secret` refresh.
- **Audit trails** – Vault audit devices stream to immutable storage with 365-day retention. ESO emits Kubernetes events for each refresh, and Loki ingests these for observability dashboards.
- **Break-glass** – In the event Vault is unavailable, disaster recover procedures allow injecting temporary secret material via sealed `SealedSecret` objects signed by security engineering. Playbook includes manual revocation steps once Vault returns.
- **Incident response** – Compromised credentials trigger rotation automations plus ACL review. Config drift detection (see `scripts/secrets/validate-stack.ts`) runs in CI and release gating.

## Deployment Automation

- Terraform module `modules/vault` provisions Vault with hardened defaults, policies, and Kubernetes auth roles.
- Terraform module `modules/external-secrets` installs ESO, the cluster secret store, and reference `ExternalSecret` templates.
- CI executes `npm run secrets:validate` to ensure documentation, Terraform manifests, and policy wiring remain in sync with security expectations.
- Playwright compliance test `tests/infra.centralized-secrets.spec.ts` prevents regressions by checking doc coverage, Terraform alignment, and automation hooks.
