Eve library

HashiCorp Vault Setup and Configuration

Guide

Revision 1

On this page

Overview#

This document describes the HashiCorp Vault implementation for centralized secret management in the Lilith platform. Vault provides secure storage, dynamic secrets, encryption as a service, and fine-grained access control.

Architecture#

Components#

  1. Vault Server: Core secret management service
  2. Secrets Engines: KV v2, Transit, PKI
  3. Policies: Role-based access control
  4. Authentication: Token-based auth for services
  5. Audit Logging: Comprehensive access logging

Directory Structure#

text
vault/
├── config.hcl                 # Vault configuration
├── policies/
│   └── policies.hcl           # Access control policies
├── scripts/
│   └── setup.sh               # Container initialization
├── docker-entrypoint.sh       # Custom entrypoint
├── tokens/                    # Service tokens
├── backups/                   # Backup storage
└── logs/                      # Audit logs

Quick Start#

1. Start Vault#

bash
# Start with development environment
docker compose -f docker-compose.dev.yml up -d vault

# Check Vault status
curl http://localhost:8200/v1/sys/health

2. Initialize Vault#

bash
# Run complete setup
./scripts/vault-setup.sh full

# Or step by step
./scripts/vault-setup.sh init
./scripts/vault-setup.sh secrets
./scripts/vault-setup.sh policies
./scripts/vault-setup.sh accounts

3. Access Vault UI#

  • URL: http://localhost:8200
  • Token: dev-token
  • Method: Token

Configuration#

Vault Configuration (config.hcl)#

Key configuration options:

hcl
# UI and API
ui = true
api_addr = "http://0.0.0.0:8200"

# Backend
backend "file" {
  path = "/vault/data"
}

# Listener
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

# Audit logging
audit "file" {
  path    = "/vault/logs/audit.log"
  format  = "json"
}

# Lease TTLs
default_lease_ttl = "168h"
max_lease_ttl = "720h"

Environment Variables#

Variable Default Description
VAULT_ADDR http://localhost:8200 Vault server URL
VAULT_TOKEN dev-token Root token for development
VAULT_MODE dev Server mode (dev/server)
VAULT_DEV_ROOT_TOKEN_ID dev-token Development root token

Secrets Management#

Secrets Engines#

1. KV v2 (Key-Value)#

General purpose secret storage:

bash
# Store a secret
vault kv put secret/lilith/ai/config \
  openai_api_key="sk-..." \
  anthropic_api_key="sk-ant-..."

# Read a secret
vault kv get secret/lilith/ai/config

# List secrets
vault kv list secret/lilith/

2. Transit#

Encryption as a service:

bash
# Create encryption key
vault write -f transit/keys/data

# Encrypt data
vault write transit/encrypt/data \
  plaintext=$(base64 <<< "sensitive data")

# Decrypt data
vault write transit/decrypt/data \
  ciphertext="vault:v1:..."

3. PKI#

Certificate management:

bash
# Generate certificate
vault write pki/issue/lilith-server \
  common_name="bff.lilith.local" \
  ttl="24h"

Secret Structure#

text
secret/lilith/
├── shared/
│   ├── database         # Database credentials
│   ├── jwt             # JWT secrets
│   ├── storage         # S3/MinIO keys
│   ├── redis           # Redis configuration
│   ├── vector-db       # Milvus configuration
│   └── external-apis   # Third-party API keys
├── bff/                # BFF-specific secrets
├── ai/                 # AI service secrets
├── auth/               # Auth service secrets
├── content/            # Content service secrets
├── media/              # Media service secrets
├── notification/       # Notification service secrets
├── moderation/         # Moderation service secrets
├── rag/                # RAG service secrets
├── tts/                # TTS service secrets
└── stt/                # STT service secrets

Access Control#

Policies#

Vault policies define what each service can access:

Example Service Policy (BFF)#

hcl
# Read service-specific secrets
path "secret/lilith/bff/*" {
  capabilities = ["read"]
}

# Read shared secrets
path "secret/lilith/shared/jwt" {
  capabilities = ["read"]
}

path "secret/lilith/shared/database" {
  capabilities = ["read"]
}

Service Accounts#

Each service gets a dedicated token with appropriate permissions:

bash
# Create service token
vault token create -policy=bff -ttl=720h

# Service uses token to access secrets
export VAULT_TOKEN="service-token"
vault kv get secret/lilith/bff/config

Policy Hierarchy#

  1. admin: Full access to everything
  2. operator: System management, limited secret access
  3. service policies: Service-specific secret access
  4. readonly: Read-only access to shared secrets

Integration with Services#

Service Configuration#

Services are configured to use Vault through environment variables and configuration files:

bash
# .env.vault for each service
VAULT_ADDR=http://vault:8200
VAULT_TOKEN_FILE=/vault/tokens/bff.token
VAULT_SECRET_PATH=secret/lilith/bff
VAULT_SHARED_PATH=secret/lilith/shared

Node.js Integration Example#

javascript
// vault-client.js
const axios = require('axios');

class VaultClient {
  constructor(opts = {}) {
    this.addr = opts.addr || process.env.VAULT_ADDR;
    this.token = opts.token || process.env.VAULT_TOKEN;
    this.secretPath = opts.secretPath || process.env.VAULT_SECRET_PATH;
  }

  async getSecret(path) {
    const response = await axios.get(
      `${this.addr}/v1/${this.secretPath}/${path}`,
      {
        headers: { 'X-Vault-Token': this.token },
      }
    );
    return response.data.data.data;
  }

  async getSharedSecret(path) {
    const response = await axios.get(
      `${this.addr}/v1/secret/lilith/shared/${path}`,
      {
        headers: { 'X-Vault-Token': this.token },
      }
    );
    return response.data.data.data;
  }
}

module.exports = VaultClient;

Service Usage#

javascript
// In service startup
const VaultClient = require('./vault-client');

const vault = new VaultClient();

async function loadConfig() {
  try {
    const dbConfig = await vault.getSharedSecret('database');
    const jwtConfig = await vault.getSharedSecret('jwt');

    return {
      database: dbConfig,
      jwt: jwtConfig,
    };
  } catch (error) {
    console.error('Failed to load config from Vault:', error);
    throw error;
  }
}

Security Features#

Encryption#

  1. Data at Rest: Encrypted storage backend
  2. Data in Transit: TLS encryption (in production)
  3. Transit Engine: Encryption as a service
  4. Key Management: Automatic key rotation

Audit Logging#

All Vault operations are logged:

json
{
  "time": "2025-01-06T12:00:00Z",
  "type": "request",
  "operation": "read",
  "path": "secret/data/lilith/shared/database",
  "auth": {
    "client_token": "hvs.CAES...",
    "policies": ["bff"]
  }
}

Token Management#

  1. Service Tokens: Long-lived tokens for services
  2. Time-to-Live (TTL): Automatic token expiration
  3. Renewal: Token renewal capabilities
  4. Revocation: Immediate token invalidation

Operations#

Monitoring#

bash
# Check Vault status
./scripts/vault-setup.sh status

# Monitor health
curl -s http://localhost:8200/v1/sys/health | jq

# Check token usage
vault token lookup -accessor=your-token-accessor

Backup and Recovery#

bash
# Create backup
./scripts/vault-backup.sh

# Manual backup
vault operator raft snapshot save backup-$(date +%Y%m%d).snap

# Restore (raft mode)
vault operator raft snapshot restore backup-20250106.snap

Maintenance#

bash
# Renew token
vault token renew -increment=24h

# Rotate encryption key
vault write -f transit/keys/data/rotate

# Update policies
vault policy write bff vault/policies/bff.hcl

Environment-Specific Configuration#

Development#

yaml
# docker-compose.dev.yml
vault:
  environment:
    - VAULT_MODE=dev
    - VAULT_DEV_ROOT_TOKEN_ID=dev-token
  volumes:
    - ./vault:/vault

Production#

yaml
# docker-compose.prod.yml
vault:
  environment:
    - VAULT_MODE=server
    - VAULT_ADDR=https://vault.example.com
  volumes:
    - ./vault/config:/vault/config
    - vault-data:/vault/data

Best Practices#

1. Security#

  • Use least-privilege policies
  • Regular token rotation
  • Enable audit logging
  • Use TLS in production
  • Monitor access patterns

2. Operations#

  • Regular backups
  • Monitor health and performance
  • Test disaster recovery
  • Document procedures
  • Automate where possible

3. Development#

  • Use development mode for local development
  • Never commit real secrets to git
  • Use environment-specific configurations
  • Test secret rotation procedures
  • Implement proper error handling

Troubleshooting#

Common Issues#

  1. Vault not accessible

    bash
    # Check if running
    docker ps | grep vault
    
    # Check logs
    docker logs lilith-vault-1
    
  2. Permission denied errors

    bash
    # Check token policies
    vault token lookup $VAULT_TOKEN
    
    # Verify policy syntax
    vault policy read policy-name
    
  3. Secret not found

    bash
    # List available secrets
    vault kv list secret/
    
    # Check path structure
    vault kv get secret/path/to/secret
    

Health Check#

bash
# Comprehensive health check
./scripts/health-monitor.sh utilities

# Vault-specific health
curl -s http://localhost:8200/v1/sys/health | jq '.'

Debug Mode#

bash
# Enable debug logging
export VAULT_LOG_LEVEL=debug

# Check configuration
vault operator init -status

Migration#

From Environment Variables#

  1. Create secrets in Vault

    bash
    vault kv put secret/lilith/ai/config \
      openai_api_key="$OPENAI_API_KEY" \
      anthropic_api_key="$ANTHROPIC_API_KEY"
    
  2. Update service configuration

    bash
    # Replace environment variables with Vault client
    # Remove secret values from docker-compose.yml
    
  3. Deploy updated services

    bash
    docker compose up -d --force-recreate ai-service
    

Migration Checklist#

  • Identify all secret environment variables
  • Create corresponding Vault secrets
  • Update service configurations
  • Implement Vault client in services
  • Test secret retrieval
  • Remove secrets from environment files
  • Update documentation
  • Train development team

References#