Eve library

Vault Migration Guide

Guide

Revision 1

On this page

Overview#

This guide describes how to migrate the Lilith platform from using environment variables for secrets to using HashiCorp Vault for secure secret management.

Why Migrate to Vault?#

Security Benefits#

  • Centralized Secret Management: All secrets stored in one secure location
  • Access Control: Fine-grained policies control who can access which secrets
  • Audit Logging: Complete audit trail of secret access
  • Dynamic Secrets: Automatic credential rotation and renewal
  • Encryption: Transit encryption for sensitive data

Operational Benefits#

  • No Hardcoded Secrets: Eliminate secrets from code and configuration files
  • Environment Consistency: Same secret management across dev, staging, and production
  • Easier Rotation: Automated secret rotation without service restarts
  • Better Developer Experience: Secure local development without exposing real secrets

Migration Process#

Prerequisites#

  1. Vault Server: Ensure Vault is running and accessible

    bash
    docker compose -f docker-compose.dev.yml up -d vault
    curl http://localhost:8200/v1/sys/health
    
  2. Initial Setup: Configure Vault with policies and secrets

    bash
    ./scripts/vault-setup.sh full
    

Migration Steps#

1. Backup Current Configuration#

bash
# Create backups before making changes
./scripts/migrate-to-vault.sh backup

2. Run Migration Script#

bash
# Complete migration
./scripts/migrate-to-vault.sh migrate

3. Validate Migration#

bash
# Test that everything works
./scripts/migrate-to-vault.sh validate

What the Migration Does#

1. Updates Docker Compose Configuration#

Before:

yaml
services:
  ai:
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - JWT_SECRET=${JWT_SECRET}

After:

yaml
services:
  ai:
    environment:
      - VAULT_ADDR=http://vault:8200
      - SERVICE_NAME=ai
      - VAULT_NAMESPACE=lilith

2. Creates Vault Client Library#

File: services/lib/vault-client-enhanced.js

Features:

  • Automatic token management
  • Caching for performance
  • Encryption/decryption support
  • Development and production modes
  • Error handling and retry logic

3. Updates Service Configuration#

Each service gets:

  • Vault Configuration File: .env.vault
  • Integration Module: src/vault-config.js
  • Startup Updates: Load config from Vault

4. Service-Specific Integrations#

AI Service#

javascript
import { loadAIConfig } from './vault-config.js';

const config = await loadAIConfig();
// Now has: config.openai_api_key, config.anthropic_api_key, etc.

Auth Service#

javascript
import { loadAuthConfig } from './vault-config.js';

const config = await loadAuthConfig();
// Now has: config.jwt_secret, config.password_salt, etc.

Notification Service#

javascript
import { loadNotificationConfig } from './vault-config.js';

const config = await loadNotificationConfig();
// Now has: config.sendgrid_api_key, config.twilio_auth_token, etc.

Post-Migration Validation#

1. Health Check#

bash
# Check Vault is accessible
curl http://localhost:8200/v1/sys/health

# Validate migration
./scripts/validate-vault-migration.sh

2. Service Testing#

bash
# Start services
docker compose -f docker-compose.dev.yml up -d

# Check service logs
docker logs lilith-ai-1
docker logs lilith-auth-1

3. Secret Access Test#

bash
# Test Vault access from services
docker exec lilith-ai-1 node -e "
const { createVaultClient } = require('./services/lib/vault-client-enhanced.js');
const vault = createVaultClient({ serviceName: 'ai' });
vault.getSecret('secret/lilith/ai/config').then(console.log);
"

Service Configuration Details#

Vault Client Configuration#

javascript
const vault = createVaultClient({
  serviceName: process.env.SERVICE_NAME, // e.g., 'ai', 'auth'
  environment: process.env.ENVIRONMENT, // 'development', 'production'
  addr: process.env.VAULT_ADDR, // 'http://vault:8200'
  token: process.env.VAULT_TOKEN, // Auto-loaded from file/env
  secretPath: 'secret/lilith/ai', // Service-specific path
  sharedPath: 'secret/lilith/shared', // Shared secrets path
  cacheTimeout: 300000, // 5 minutes
});

Secret Structure in Vault#

text
secret/lilith/
├── shared/
│   ├── database          # Database credentials
│   ├── jwt              # JWT secrets
│   ├── storage          # S3/MinIO keys
│   ├── redis            # Redis configuration
│   └── external-apis    # Third-party API keys
├── ai/
│   └── config           # AI service secrets
├── auth/
│   └── config           # Auth service secrets
├── notification/
│   └── config           # Notification service secrets
└── media/
    └── config           # Media service secrets

Access Patterns#

Shared Secrets#

javascript
// Any service can access shared secrets
const dbConfig = await vault.getSharedSecret('database');
const jwtConfig = await vault.getSharedSecret('jwt');

Service-Specific Secrets#

javascript
// Only the AI service can access its secrets
const aiConfig = await vault.getServiceSecret('config');

Direct Path Access#

javascript
// Direct access to any secret
const secret = await vault.getSecret('secret/lilith/shared/database');

Security Considerations#

1. Token Management#

  • Service Tokens: Long-lived tokens for services
  • Human Tokens: Short-lived tokens for developers
  • Auto-Renewal: Automatic token renewal before expiration

2. Access Control#

  • Least Privilege: Services only access secrets they need
  • Policy Enforcement: Vault policies prevent unauthorized access
  • Audit Logging: All access logged and monitored

3. Secret Rotation#

  • Automated Rotation: Configure automatic secret rotation
  • Versioning: Keep multiple versions of secrets
  • Rollback: Ability to rollback to previous secret versions

Development Workflow#

1. Local Development#

bash
# Start Vault in development mode
docker compose -f docker-compose.dev.yml up -d vault

# Setup initial secrets
./scripts/vault-setup.sh secrets

# Work with services normally
docker compose -f docker-compose.dev.yml up -d ai

2. Adding New Secrets#

Via Vault UI:

  1. Open http://localhost:8200
  2. Navigate to secret/
  3. Add new secret under appropriate path

Via CLI:

bash
# Add shared secret
vault kv put secret/lilith/shared/new-secret \
  api_key="your-api-key" \
  endpoint="https://api.example.com"

# Add service-specific secret
vault kv put secret/lilith/ai/new-model \
  model_name="gpt-4-turbo" \
  api_key="your-model-key"

Via Script:

bash
# Use setup script
./scripts/vault-setup.sh manage

3. Testing New Secrets#

javascript
// In service code
const newSecret = await vault.getSharedSecret('new-secret');
console.log(newSecret.api_key);

Production Considerations#

1. Vault Configuration#

yaml
# docker-compose.prod.yml
vault:
  environment:
    - VAULT_MODE=server
    - VAULT_ADDR=https://vault.example.com
    - VAULT_TLS_CERT_FILE=/vault/tls/cert.pem
    - VAULT_TLS_KEY_FILE=/vault/tls/key.pem
  volumes:
    - ./vault/config:/vault/config
    - vault-data:/vault/data
    - ./vault/tls:/vault/tls

2. Service Configuration#

bash
# Production environment variables
export VAULT_ADDR=https://vault.example.com
export VAULT_NAMESPACE=lilith
export VAULT_TLS_SKIP_VERIFY=false

3. Authentication Methods#

Kubernetes Auth:

javascript
// Service authenticates with Kubernetes service account
const vault = createVaultClient({
  authMethod: 'kubernetes',
  roleName: 'ai-service',
});

await vault.authenticateK8s();

AWS IAM Auth:

javascript
// Service authenticates with AWS IAM
const vault = createVaultClient({
  authMethod: 'aws-iam',
  role: 'ai-service',
});

Troubleshooting#

Common Issues#

1. Service Cannot Connect to Vault#

bash
# Check Vault connectivity
docker exec lilith-ai-1 curl -f http://vault:8200/v1/sys/health

# Check network
docker network ls | grep lilith

2. Token Authentication Failed#

bash
# Check token validity
vault token lookup $VAULT_TOKEN

# Create new service token
./scripts/vault-setup.sh accounts

3. Secret Not Found#

bash
# Check if secret exists
vault kv get secret/lilith/shared/database

# List available secrets
vault kv list secret/lilith/

4. Permission Denied#

bash
# Check service policies
vault policy read ai

# Verify token permissions
vault token lookup -accessor=$TOKEN_ACCESSOR

Debug Mode#

Enable debug logging:

bash
# Set environment variable
export VAULT_LOG_LEVEL=debug

# Or update service configuration
export DEBUG=vault:*

Cache Issues#

Clear Vault client cache:

javascript
// In service code
vault.clearCache();

Best Practices#

1. Development#

  • Never commit secrets to version control
  • Use development tokens for local development
  • Test secret rotation procedures regularly
  • Validate access patterns in staging

2. Production#

  • Use TLS for all Vault communication
  • Enable audit logging and monitor regularly
  • Implement automatic token renewal
  • Regular security audits of Vault policies

3. Operations#

  • Backup Vault data regularly
  • Test disaster recovery procedures
  • Monitor Vault performance and health
  • Document all procedures and configurations

Rollback Plan#

If migration fails or causes issues:

1. Stop Services#

bash
docker compose -f docker-compose.dev.yml down

2. Restore Backups#

bash
# Find latest backup
ls -la backups/vault-migration-*

# Restore docker-compose.dev.yml
cp backups/vault-migration-*/docker-compose.dev.yml .

3. Restart Services#

bash
docker compose -f docker-compose.dev.yml up -d

4. Validate#

bash
# Check services are working with old config
docker compose -f docker-compose.dev.yml ps

Monitoring and Maintenance#

1. Vault Health Monitoring#

bash
# Health check script
./scripts/health-monitor.sh utilities

# Vault-specific metrics
curl -s http://localhost:8200/v1/sys/metrics | jq

2. Secret Usage Monitoring#

bash
# Check token usage
vault token lookup -accessor=$TOKEN_ACCESSOR

# Audit log monitoring
tail -f vault/logs/audit.log | jq '.request.path'

3. Performance Monitoring#

bash
# Monitor response times
curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8200/v1/sys/health

# Monitor cache hit rates
# (Add metrics to your vault client)

Next Steps#

After migration:

  1. Test Thoroughly: Ensure all services work with Vault
  2. Monitor Performance: Check for any performance impact
  3. Train Team: Ensure team members understand Vault usage
  4. Update Documentation: Keep all documentation current
  5. Regular Maintenance: Schedule regular secret rotation and cleanup

Support#

  • Documentation: See docs/deploy/VAULT_SETUP.md
  • Scripts: See scripts/vault-setup.sh
  • Health Monitoring: See scripts/health-monitor.sh
  • Issues: Create GitHub issues for problems