Overview#
This guide describes how to migrate the Lilith platform from using environment variables for secrets to using HashiCorp Vault for secure secret management.
Why Migrate to Vault?#
Security Benefits#
- Centralized Secret Management: All secrets stored in one secure location
- Access Control: Fine-grained policies control who can access which secrets
- Audit Logging: Complete audit trail of secret access
- Dynamic Secrets: Automatic credential rotation and renewal
- Encryption: Transit encryption for sensitive data
Operational Benefits#
- No Hardcoded Secrets: Eliminate secrets from code and configuration files
- Environment Consistency: Same secret management across dev, staging, and production
- Easier Rotation: Automated secret rotation without service restarts
- Better Developer Experience: Secure local development without exposing real secrets
Migration Process#
Prerequisites#
-
Vault Server: Ensure Vault is running and accessible
bashdocker compose -f docker-compose.dev.yml up -d vault curl http://localhost:8200/v1/sys/health -
Initial Setup: Configure Vault with policies and secrets
bash./scripts/vault-setup.sh full
Migration Steps#
1. Backup Current Configuration#
bash
# Create backups before making changes
./scripts/migrate-to-vault.sh backup
2. Run Migration Script#
bash
# Complete migration
./scripts/migrate-to-vault.sh migrate
3. Validate Migration#
bash
# Test that everything works
./scripts/migrate-to-vault.sh validate
What the Migration Does#
1. Updates Docker Compose Configuration#
Before:
yaml
services:
ai:
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- JWT_SECRET=${JWT_SECRET}
After:
yaml
services:
ai:
environment:
- VAULT_ADDR=http://vault:8200
- SERVICE_NAME=ai
- VAULT_NAMESPACE=lilith
2. Creates Vault Client Library#
File: services/lib/vault-client-enhanced.js
Features:
- Automatic token management
- Caching for performance
- Encryption/decryption support
- Development and production modes
- Error handling and retry logic
3. Updates Service Configuration#
Each service gets:
- Vault Configuration File:
.env.vault - Integration Module:
src/vault-config.js - Startup Updates: Load config from Vault
4. Service-Specific Integrations#
AI Service#
javascript
import { loadAIConfig } from './vault-config.js';
const config = await loadAIConfig();
// Now has: config.openai_api_key, config.anthropic_api_key, etc.
Auth Service#
javascript
import { loadAuthConfig } from './vault-config.js';
const config = await loadAuthConfig();
// Now has: config.jwt_secret, config.password_salt, etc.
Notification Service#
javascript
import { loadNotificationConfig } from './vault-config.js';
const config = await loadNotificationConfig();
// Now has: config.sendgrid_api_key, config.twilio_auth_token, etc.
Post-Migration Validation#
1. Health Check#
bash
# Check Vault is accessible
curl http://localhost:8200/v1/sys/health
# Validate migration
./scripts/validate-vault-migration.sh
2. Service Testing#
bash
# Start services
docker compose -f docker-compose.dev.yml up -d
# Check service logs
docker logs lilith-ai-1
docker logs lilith-auth-1
3. Secret Access Test#
bash
# Test Vault access from services
docker exec lilith-ai-1 node -e "
const { createVaultClient } = require('./services/lib/vault-client-enhanced.js');
const vault = createVaultClient({ serviceName: 'ai' });
vault.getSecret('secret/lilith/ai/config').then(console.log);
"
Service Configuration Details#
Vault Client Configuration#
javascript
const vault = createVaultClient({
serviceName: process.env.SERVICE_NAME, // e.g., 'ai', 'auth'
environment: process.env.ENVIRONMENT, // 'development', 'production'
addr: process.env.VAULT_ADDR, // 'http://vault:8200'
token: process.env.VAULT_TOKEN, // Auto-loaded from file/env
secretPath: 'secret/lilith/ai', // Service-specific path
sharedPath: 'secret/lilith/shared', // Shared secrets path
cacheTimeout: 300000, // 5 minutes
});
Secret Structure in Vault#
text
secret/lilith/
├── shared/
│ ├── database # Database credentials
│ ├── jwt # JWT secrets
│ ├── storage # S3/MinIO keys
│ ├── redis # Redis configuration
│ └── external-apis # Third-party API keys
├── ai/
│ └── config # AI service secrets
├── auth/
│ └── config # Auth service secrets
├── notification/
│ └── config # Notification service secrets
└── media/
└── config # Media service secrets
Access Patterns#
Shared Secrets#
javascript
// Any service can access shared secrets
const dbConfig = await vault.getSharedSecret('database');
const jwtConfig = await vault.getSharedSecret('jwt');
Service-Specific Secrets#
javascript
// Only the AI service can access its secrets
const aiConfig = await vault.getServiceSecret('config');
Direct Path Access#
javascript
// Direct access to any secret
const secret = await vault.getSecret('secret/lilith/shared/database');
Security Considerations#
1. Token Management#
- Service Tokens: Long-lived tokens for services
- Human Tokens: Short-lived tokens for developers
- Auto-Renewal: Automatic token renewal before expiration
2. Access Control#
- Least Privilege: Services only access secrets they need
- Policy Enforcement: Vault policies prevent unauthorized access
- Audit Logging: All access logged and monitored
3. Secret Rotation#
- Automated Rotation: Configure automatic secret rotation
- Versioning: Keep multiple versions of secrets
- Rollback: Ability to rollback to previous secret versions
Development Workflow#
1. Local Development#
bash
# Start Vault in development mode
docker compose -f docker-compose.dev.yml up -d vault
# Setup initial secrets
./scripts/vault-setup.sh secrets
# Work with services normally
docker compose -f docker-compose.dev.yml up -d ai
2. Adding New Secrets#
Via Vault UI:
- Open http://localhost:8200
- Navigate to
secret/ - Add new secret under appropriate path
Via CLI:
bash
# Add shared secret
vault kv put secret/lilith/shared/new-secret \
api_key="your-api-key" \
endpoint="https://api.example.com"
# Add service-specific secret
vault kv put secret/lilith/ai/new-model \
model_name="gpt-4-turbo" \
api_key="your-model-key"
Via Script:
bash
# Use setup script
./scripts/vault-setup.sh manage
3. Testing New Secrets#
javascript
// In service code
const newSecret = await vault.getSharedSecret('new-secret');
console.log(newSecret.api_key);
Production Considerations#
1. Vault Configuration#
yaml
# docker-compose.prod.yml
vault:
environment:
- VAULT_MODE=server
- VAULT_ADDR=https://vault.example.com
- VAULT_TLS_CERT_FILE=/vault/tls/cert.pem
- VAULT_TLS_KEY_FILE=/vault/tls/key.pem
volumes:
- ./vault/config:/vault/config
- vault-data:/vault/data
- ./vault/tls:/vault/tls
2. Service Configuration#
bash
# Production environment variables
export VAULT_ADDR=https://vault.example.com
export VAULT_NAMESPACE=lilith
export VAULT_TLS_SKIP_VERIFY=false
3. Authentication Methods#
Kubernetes Auth:
javascript
// Service authenticates with Kubernetes service account
const vault = createVaultClient({
authMethod: 'kubernetes',
roleName: 'ai-service',
});
await vault.authenticateK8s();
AWS IAM Auth:
javascript
// Service authenticates with AWS IAM
const vault = createVaultClient({
authMethod: 'aws-iam',
role: 'ai-service',
});
Troubleshooting#
Common Issues#
1. Service Cannot Connect to Vault#
bash
# Check Vault connectivity
docker exec lilith-ai-1 curl -f http://vault:8200/v1/sys/health
# Check network
docker network ls | grep lilith
2. Token Authentication Failed#
bash
# Check token validity
vault token lookup $VAULT_TOKEN
# Create new service token
./scripts/vault-setup.sh accounts
3. Secret Not Found#
bash
# Check if secret exists
vault kv get secret/lilith/shared/database
# List available secrets
vault kv list secret/lilith/
4. Permission Denied#
bash
# Check service policies
vault policy read ai
# Verify token permissions
vault token lookup -accessor=$TOKEN_ACCESSOR
Debug Mode#
Enable debug logging:
bash
# Set environment variable
export VAULT_LOG_LEVEL=debug
# Or update service configuration
export DEBUG=vault:*
Cache Issues#
Clear Vault client cache:
javascript
// In service code
vault.clearCache();
Best Practices#
1. Development#
- Never commit secrets to version control
- Use development tokens for local development
- Test secret rotation procedures regularly
- Validate access patterns in staging
2. Production#
- Use TLS for all Vault communication
- Enable audit logging and monitor regularly
- Implement automatic token renewal
- Regular security audits of Vault policies
3. Operations#
- Backup Vault data regularly
- Test disaster recovery procedures
- Monitor Vault performance and health
- Document all procedures and configurations
Rollback Plan#
If migration fails or causes issues:
1. Stop Services#
bash
docker compose -f docker-compose.dev.yml down
2. Restore Backups#
bash
# Find latest backup
ls -la backups/vault-migration-*
# Restore docker-compose.dev.yml
cp backups/vault-migration-*/docker-compose.dev.yml .
3. Restart Services#
bash
docker compose -f docker-compose.dev.yml up -d
4. Validate#
bash
# Check services are working with old config
docker compose -f docker-compose.dev.yml ps
Monitoring and Maintenance#
1. Vault Health Monitoring#
bash
# Health check script
./scripts/health-monitor.sh utilities
# Vault-specific metrics
curl -s http://localhost:8200/v1/sys/metrics | jq
2. Secret Usage Monitoring#
bash
# Check token usage
vault token lookup -accessor=$TOKEN_ACCESSOR
# Audit log monitoring
tail -f vault/logs/audit.log | jq '.request.path'
3. Performance Monitoring#
bash
# Monitor response times
curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8200/v1/sys/health
# Monitor cache hit rates
# (Add metrics to your vault client)
Next Steps#
After migration:
- Test Thoroughly: Ensure all services work with Vault
- Monitor Performance: Check for any performance impact
- Train Team: Ensure team members understand Vault usage
- Update Documentation: Keep all documentation current
- Regular Maintenance: Schedule regular secret rotation and cleanup
Support#
- Documentation: See
docs/deploy/VAULT_SETUP.md - Scripts: See
scripts/vault-setup.sh - Health Monitoring: See
scripts/health-monitor.sh - Issues: Create GitHub issues for problems