Source of truth for every Vault operational task. All configuration lives in
infrastructure/vault/; nothing under deploy/ is canonical any more.
Prerequisites#
vaultCLI (>= 1.15) on your workstation or in a break-glass podkubectlwith cluster-admin on the target cluster- AWS credentials with
kms:Encrypt/kms:Decrypton the unseal key (prod only) - Postgres superuser credentials exported as
POSTGRES_SUPERPASS(bootstrap uses these to provision the database engine root, then rotates it)
Install order#
1. Deploy the Vault StatefulSet#
kubectl apply -f infrastructure/vault/kubernetes/namespace.yaml
kubectl apply -f infrastructure/vault/kubernetes/serviceaccount.yaml
kubectl apply -f infrastructure/vault/kubernetes/service.yaml
kubectl apply -f infrastructure/vault/kubernetes/statefulset.yaml
kubectl apply -f infrastructure/vault/kubernetes/ingress.yaml
kubectl apply -f infrastructure/vault/kubernetes/servicemonitor.yaml
Alternatively, install the upstream Vault Helm chart with
infrastructure/vault/kubernetes/vault-agent-injector/values.yaml to layer the
Agent Injector on top.
2. Initialize the first Raft peer#
kubectl -n vault exec -it vault-0 -- vault operator init \
-recovery-shares=5 -recovery-threshold=3 -format=json > vault-init.json
vault-init.json contains the recovery keys and the initial root token. Split
recovery keys across operators and delete the file. With AWS KMS auto-unseal,
sealing happens automatically on restart; recovery keys are only used to
regenerate root or rotate the seal.
3. Join the other peers to Raft#
for p in 1 2; do
kubectl -n vault exec -it vault-$p -- vault operator raft join \
https://vault-0.vault-internal:8200
done
4. Run bootstrap#
export VAULT_ADDR=https://vault.oshun.io:8200
export VAULT_TOKEN=$(jq -r '.root_token' vault-init.json)
export KUBERNETES_HOST=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
export POSTGRES_HOST=postgres.oshun-data.svc.cluster.local
export POSTGRES_SUPERUSER=vault_root
export POSTGRES_SUPERPASS='<superuser password>'
export OSHUN_ENV=prod
./infrastructure/vault/bootstrap/bootstrap.sh
The script is idempotent; re-run it safely whenever you change a policy or add a new domain.
5. Revoke the root token#
vault token revoke "$VAULT_TOKEN"
unset VAULT_TOKEN
Day-to-day operators authenticate via OIDC/LDAP mapped to the admin policy
(configure the method of your choice separately).
Sealing and unsealing#
Auto-unseal is driven by AWS KMS in prod. For dev (config.dev.hcl) Vault
starts in -dev mode with a static root token from VAULT_DEV_ROOT_TOKEN_ID.
Emergency reseal:
vault operator seal
Recovery-key-assisted root regeneration (break-glass):
vault operator generate-root -init
# Each recovery-key holder runs:
vault operator generate-root -otp=<nonce>
Rotation#
| Artifact | Cadence | Mechanism |
|---|---|---|
| Transit key versions | 90 days (auto) | auto_rotate_period=2160h on each transit/keys/<domain> |
| DB root credential | 30 days | vault write -force database/rotate-root/<domain>-postgres |
| Dynamic DB creds | 1h TTL (app) | Vault leases; no action needed |
| CI approle secret-id | 24h TTL | vault write -f auth/approle/role/ci/secret-id via nightly workflow |
| KV secrets (per-leaf) | Per domain SLA | vault kv put secret/<domain>/... with CI/audit trail |
| Recovery shares | On operator exit | vault operator generate-root + re-shard |
| Seal (KMS key) | Annually | vault operator seal -> rotate KMS -> re-init with new seal block |
Observability#
-
Prometheus scrapes
/v1/sys/metrics?format=prometheusvia the ServiceMonitor ininfrastructure/vault/kubernetes/servicemonitor.yaml. A bearer token with onlyreadonsys/metricsis required; mint it with:bashvault policy write vault-metrics - <<'POL' path "sys/metrics" { capabilities = ["read"] } POL vault token create -policy=vault-metrics -ttl=8760h -renewable=true -field=token \ > /tmp/vault-metrics-token kubectl -n oshun-monitoring create secret generic vault-metrics-token \ --from-file=token=/tmp/vault-metrics-token -
Audit log lives at
/vault/audit/audit.loginside each pod. The recommended shipping pipeline is a sidecar that tails the file into your SIEM.
Disaster recovery#
-
Snapshot the Raft state daily:
bashvault operator raft snapshot save /tmp/vault-$(date +%F).snap -
Restore:
bashvault operator raft snapshot restore /tmp/vault-2026-04-18.snap -
If the KMS key is lost, use the recovery keys from step 2 of install to regenerate root and re-seal against a new KMS key.
Domain addition#
When onboarding a new domain:
- Add
policies/<new>.hclfollowing the shape oflilith.hcl. - Append the domain to
DOMAINS=(...)inbootstrap.sh. - Add Terraform wiring in
infrastructure/terraform/modules/vault/. - Create
infrastructure/vault/kubernetes/external-secrets/<new>.yaml. - Re-run
bootstrap.sh.
References#
- Legacy configs retained (do not edit) under
infra/vault-deploy/,infra/yemaya/vault/,infra/yemaya/docker/vault-agent/, anddeploy/terraform/modules/vault/. - Secret path layout:
infrastructure/vault/SECRET_PATHS.md.