Eve library

Oshun Vault Operator Runbook

Other

Revision 1

On this page

Source of truth for every Vault operational task. All configuration lives in infrastructure/vault/; nothing under deploy/ is canonical any more.

Prerequisites#

  • vault CLI (>= 1.15) on your workstation or in a break-glass pod
  • kubectl with cluster-admin on the target cluster
  • AWS credentials with kms:Encrypt/kms:Decrypt on the unseal key (prod only)
  • Postgres superuser credentials exported as POSTGRES_SUPERPASS (bootstrap uses these to provision the database engine root, then rotates it)

Install order#

1. Deploy the Vault StatefulSet#

bash
kubectl apply -f infrastructure/vault/kubernetes/namespace.yaml
kubectl apply -f infrastructure/vault/kubernetes/serviceaccount.yaml
kubectl apply -f infrastructure/vault/kubernetes/service.yaml
kubectl apply -f infrastructure/vault/kubernetes/statefulset.yaml
kubectl apply -f infrastructure/vault/kubernetes/ingress.yaml
kubectl apply -f infrastructure/vault/kubernetes/servicemonitor.yaml

Alternatively, install the upstream Vault Helm chart with infrastructure/vault/kubernetes/vault-agent-injector/values.yaml to layer the Agent Injector on top.

2. Initialize the first Raft peer#

bash
kubectl -n vault exec -it vault-0 -- vault operator init \
  -recovery-shares=5 -recovery-threshold=3 -format=json > vault-init.json

vault-init.json contains the recovery keys and the initial root token. Split recovery keys across operators and delete the file. With AWS KMS auto-unseal, sealing happens automatically on restart; recovery keys are only used to regenerate root or rotate the seal.

3. Join the other peers to Raft#

bash
for p in 1 2; do
  kubectl -n vault exec -it vault-$p -- vault operator raft join \
    https://vault-0.vault-internal:8200
done

4. Run bootstrap#

bash
export VAULT_ADDR=https://vault.oshun.io:8200
export VAULT_TOKEN=$(jq -r '.root_token' vault-init.json)
export KUBERNETES_HOST=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
export POSTGRES_HOST=postgres.oshun-data.svc.cluster.local
export POSTGRES_SUPERUSER=vault_root
export POSTGRES_SUPERPASS='<superuser password>'
export OSHUN_ENV=prod

./infrastructure/vault/bootstrap/bootstrap.sh

The script is idempotent; re-run it safely whenever you change a policy or add a new domain.

5. Revoke the root token#

bash
vault token revoke "$VAULT_TOKEN"
unset VAULT_TOKEN

Day-to-day operators authenticate via OIDC/LDAP mapped to the admin policy (configure the method of your choice separately).

Sealing and unsealing#

Auto-unseal is driven by AWS KMS in prod. For dev (config.dev.hcl) Vault starts in -dev mode with a static root token from VAULT_DEV_ROOT_TOKEN_ID.

Emergency reseal:

bash
vault operator seal

Recovery-key-assisted root regeneration (break-glass):

bash
vault operator generate-root -init
# Each recovery-key holder runs:
vault operator generate-root -otp=<nonce>

Rotation#

Artifact Cadence Mechanism
Transit key versions 90 days (auto) auto_rotate_period=2160h on each transit/keys/<domain>
DB root credential 30 days vault write -force database/rotate-root/<domain>-postgres
Dynamic DB creds 1h TTL (app) Vault leases; no action needed
CI approle secret-id 24h TTL vault write -f auth/approle/role/ci/secret-id via nightly workflow
KV secrets (per-leaf) Per domain SLA vault kv put secret/<domain>/... with CI/audit trail
Recovery shares On operator exit vault operator generate-root + re-shard
Seal (KMS key) Annually vault operator seal -> rotate KMS -> re-init with new seal block

Observability#

  • Prometheus scrapes /v1/sys/metrics?format=prometheus via the ServiceMonitor in infrastructure/vault/kubernetes/servicemonitor.yaml. A bearer token with only read on sys/metrics is required; mint it with:

    bash
    vault policy write vault-metrics - <<'POL'
    path "sys/metrics" { capabilities = ["read"] }
    POL
    vault token create -policy=vault-metrics -ttl=8760h -renewable=true -field=token \
      > /tmp/vault-metrics-token
    kubectl -n oshun-monitoring create secret generic vault-metrics-token \
      --from-file=token=/tmp/vault-metrics-token
    
  • Audit log lives at /vault/audit/audit.log inside each pod. The recommended shipping pipeline is a sidecar that tails the file into your SIEM.

Disaster recovery#

  1. Snapshot the Raft state daily:

    bash
    vault operator raft snapshot save /tmp/vault-$(date +%F).snap
    
  2. Restore:

    bash
    vault operator raft snapshot restore /tmp/vault-2026-04-18.snap
    
  3. If the KMS key is lost, use the recovery keys from step 2 of install to regenerate root and re-seal against a new KMS key.

Domain addition#

When onboarding a new domain:

  1. Add policies/<new>.hcl following the shape of lilith.hcl.
  2. Append the domain to DOMAINS=(...) in bootstrap.sh.
  3. Add Terraform wiring in infrastructure/terraform/modules/vault/.
  4. Create infrastructure/vault/kubernetes/external-secrets/<new>.yaml.
  5. Re-run bootstrap.sh.

References#

  • Legacy configs retained (do not edit) under infra/vault-deploy/, infra/yemaya/vault/, infra/yemaya/docker/vault-agent/, and deploy/terraform/modules/vault/.
  • Secret path layout: infrastructure/vault/SECRET_PATHS.md.