# Oshun Vault Operator Runbook

Source of truth for every Vault operational task. All configuration lives in
`infrastructure/vault/`; nothing under `deploy/` is canonical any more.

## Prerequisites

- `vault` CLI (>= 1.15) on your workstation or in a break-glass pod
- `kubectl` with cluster-admin on the target cluster
- AWS credentials with `kms:Encrypt`/`kms:Decrypt` on the unseal key (prod only)
- Postgres superuser credentials exported as `POSTGRES_SUPERPASS` (bootstrap
  uses these to provision the database engine root, then rotates it)

## Install order

### 1. Deploy the Vault StatefulSet

```bash
kubectl apply -f infrastructure/vault/kubernetes/namespace.yaml
kubectl apply -f infrastructure/vault/kubernetes/serviceaccount.yaml
kubectl apply -f infrastructure/vault/kubernetes/service.yaml
kubectl apply -f infrastructure/vault/kubernetes/statefulset.yaml
kubectl apply -f infrastructure/vault/kubernetes/ingress.yaml
kubectl apply -f infrastructure/vault/kubernetes/servicemonitor.yaml
```

Alternatively, install the upstream Vault Helm chart with
`infrastructure/vault/kubernetes/vault-agent-injector/values.yaml` to layer the
Agent Injector on top.

### 2. Initialize the first Raft peer

```bash
kubectl -n vault exec -it vault-0 -- vault operator init \
  -recovery-shares=5 -recovery-threshold=3 -format=json > vault-init.json
```

`vault-init.json` contains the recovery keys and the initial root token. **Split
recovery keys across operators and delete the file.** With AWS KMS auto-unseal,
sealing happens automatically on restart; recovery keys are only used to
regenerate root or rotate the seal.

### 3. Join the other peers to Raft

```bash
for p in 1 2; do
  kubectl -n vault exec -it vault-$p -- vault operator raft join \
    https://vault-0.vault-internal:8200
done
```

### 4. Run bootstrap

```bash
export VAULT_ADDR=https://vault.oshun.io:8200
export VAULT_TOKEN=$(jq -r '.root_token' vault-init.json)
export KUBERNETES_HOST=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
export POSTGRES_HOST=postgres.oshun-data.svc.cluster.local
export POSTGRES_SUPERUSER=vault_root
export POSTGRES_SUPERPASS='<superuser password>'
export OSHUN_ENV=prod

./infrastructure/vault/bootstrap/bootstrap.sh
```

The script is idempotent; re-run it safely whenever you change a policy or add a
new domain.

### 5. Revoke the root token

```bash
vault token revoke "$VAULT_TOKEN"
unset VAULT_TOKEN
```

Day-to-day operators authenticate via OIDC/LDAP mapped to the `admin` policy
(configure the method of your choice separately).

## Sealing and unsealing

Auto-unseal is driven by AWS KMS in prod. For dev (`config.dev.hcl`) Vault
starts in `-dev` mode with a static root token from `VAULT_DEV_ROOT_TOKEN_ID`.

Emergency reseal:

```bash
vault operator seal
```

Recovery-key-assisted root regeneration (break-glass):

```bash
vault operator generate-root -init
# Each recovery-key holder runs:
vault operator generate-root -otp=<nonce>
```

## Rotation

| Artifact              | Cadence          | Mechanism                                                            |
| --------------------- | ---------------- | -------------------------------------------------------------------- |
| Transit key versions  | 90 days (auto)   | `auto_rotate_period=2160h` on each transit/keys/<domain>             |
| DB root credential    | 30 days          | `vault write -force database/rotate-root/<domain>-postgres`          |
| Dynamic DB creds      | 1h TTL (app)     | Vault leases; no action needed                                       |
| CI approle secret-id  | 24h TTL          | `vault write -f auth/approle/role/ci/secret-id` via nightly workflow |
| KV secrets (per-leaf) | Per domain SLA   | `vault kv put secret/<domain>/...` with CI/audit trail               |
| Recovery shares       | On operator exit | `vault operator generate-root` + re-shard                            |
| Seal (KMS key)        | Annually         | `vault operator seal` -> rotate KMS -> re-init with new seal block   |

## Observability

- Prometheus scrapes `/v1/sys/metrics?format=prometheus` via the ServiceMonitor
  in `infrastructure/vault/kubernetes/servicemonitor.yaml`. A bearer token with
  only `read` on `sys/metrics` is required; mint it with:

  ```bash
  vault policy write vault-metrics - <<'POL'
  path "sys/metrics" { capabilities = ["read"] }
  POL
  vault token create -policy=vault-metrics -ttl=8760h -renewable=true -field=token \
    > /tmp/vault-metrics-token
  kubectl -n oshun-monitoring create secret generic vault-metrics-token \
    --from-file=token=/tmp/vault-metrics-token
  ```

- Audit log lives at `/vault/audit/audit.log` inside each pod. The recommended
  shipping pipeline is a sidecar that tails the file into your SIEM.

## Disaster recovery

1. Snapshot the Raft state daily:

   ```bash
   vault operator raft snapshot save /tmp/vault-$(date +%F).snap
   ```

2. Restore:

   ```bash
   vault operator raft snapshot restore /tmp/vault-2026-04-18.snap
   ```

3. If the KMS key is lost, use the recovery keys from step 2 of install to
   regenerate root and re-seal against a new KMS key.

## Domain addition

When onboarding a new domain:

1. Add `policies/<new>.hcl` following the shape of `lilith.hcl`.
2. Append the domain to `DOMAINS=(...)` in `bootstrap.sh`.
3. Add Terraform wiring in `infrastructure/terraform/modules/vault/`.
4. Create `infrastructure/vault/kubernetes/external-secrets/<new>.yaml`.
5. Re-run `bootstrap.sh`.

## References

- Legacy configs retained (do not edit) under `infra/vault-deploy/`,
  `infra/yemaya/vault/`, `infra/yemaya/docker/vault-agent/`, and
  `deploy/terraform/modules/vault/`.
- Secret path layout: `infrastructure/vault/SECRET_PATHS.md`.
