Lilith standardizes on HashiCorp Vault as the control plane for runtime configuration and sensitive material, with External Secrets Operator (ESO) bridging secrets into Kubernetes namespaces and the ConfigManager library enforcing schema validation inside each service. This document captures architecture decisions, operational procedures, and guardrails that keep credentials safe, rotated, and observable.
Vault Architecture and Hardening#
- Deployment topology – Vault runs as a highly available cluster (Raft storage + AWS KMS auto-unseal) in namespace
vault. Helm values enforce TLS everywhere, dedicated storage classes, and separate I/O classes for audit and data volumes. - Network policy – Only in-cluster callers in the same namespace may reach Vault’s HTTPS interface. Egress is restricted to DNS + AWS APIs for KMS interactions.
- Authentication – Kubernetes auth backend is enabled with dedicated roles per service account (
bff,auth,ai, etc.) plus a sharedexternal-secretsrole for ESO. Tokens are scoped to the minimum TTL required (1 hour) and are renewable via the Vault client. - Secret engines – KV v2 is mounted at
secret/for configuration payloads, and the database engine underdatabase/issues short-lived credentials for Postgres, Milvus, and analytics replicas. Audit logging is enabled with immutability enforced through remote storage. - Policies – Policy files under
deploy/terraform/modules/vault/policiesdescribe explicit capability lists. ConfigMap-style reads live in read-only paths, while dynamic credentials are exposed via narrow prefixes so leaking one policy cannot escalate to another service.
External Secrets Operator Integration#
- Operator – Terraform installs the ESO Helm chart in namespace
external-secrets, enabling CRDs, RBAC, and a service account bound to theexternal-secretsVault role. - ClusterSecretStore – A cluster-wide store points at
https://vault.vault.svc.cluster.local:8200, uses Kubernetes auth, and reuses the Vault CA bundle from thevault-tlssecret. - Sync workflow –
ExternalSecretresources declare which Vault paths should materialize as native KubernetesSecretobjects. A defaultapp-configmapping fans out base configuration (database URLs, feature flags) into the application namespace while leaving ownership inside Vault. - Rotation cadence – ESO reconciles every 60 seconds by default. When
SecretRotationManagerrotates material inside Vault, the next reconcile updates dependent secrets automatically without pods restarting.
Configuration Distribution & Schema Enforcement#
- ConfigManager – Services consume
services/lib/config-manager.js, which loads hierarchical config (base → environment → service overrides) and validates against JSON Schema. Vault values take precedence over filesystem defaults. - External Credential Manager – Provider keys (OpenAI, Anthropic, AWS) live under
secret/providers/*. ConfigManager dynamically injects them into the service container, while masking occurs before logs or metrics ingestion. - Environment parity – The Terraform stack publishes
ClusterSecretStore,ExternalSecret, and namespace wiring per environment. CI verifies manifests so staging/prod drift cannot occur.
Rotation, Auditing, and Break-glass Procedures#
- Rotation automation –
SecretRotationManagerregisters rotation hooks per secret. Database credentials rotate every 24h, API keys every 7d, and TLS certificates every 30d. ESO propagates fresh material to pods throughSecretrefresh. - Audit trails – Vault audit devices stream to immutable storage with 365-day retention. ESO emits Kubernetes events for each refresh, and Loki ingests these for observability dashboards.
- Break-glass – In the event Vault is unavailable, disaster recover procedures allow injecting temporary secret material via sealed
SealedSecretobjects signed by security engineering. Playbook includes manual revocation steps once Vault returns. - Incident response – Compromised credentials trigger rotation automations plus ACL review. Config drift detection (see
scripts/secrets/validate-stack.ts) runs in CI and release gating.
Deployment Automation#
- Terraform module
modules/vaultprovisions Vault with hardened defaults, policies, and Kubernetes auth roles. - Terraform module
modules/external-secretsinstalls ESO, the cluster secret store, and referenceExternalSecrettemplates. - CI executes
npm run secrets:validateto ensure documentation, Terraform manifests, and policy wiring remain in sync with security expectations. - Playwright compliance test
tests/infra.centralized-secrets.spec.tsprevents regressions by checking doc coverage, Terraform alignment, and automation hooks.