# Vault Migration Guide

## Overview

This guide describes how to migrate the Lilith platform from using environment
variables for secrets to using HashiCorp Vault for secure secret management.

## Why Migrate to Vault?

### Security Benefits

- **Centralized Secret Management**: All secrets stored in one secure location
- **Access Control**: Fine-grained policies control who can access which secrets
- **Audit Logging**: Complete audit trail of secret access
- **Dynamic Secrets**: Automatic credential rotation and renewal
- **Encryption**: Transit encryption for sensitive data

### Operational Benefits

- **No Hardcoded Secrets**: Eliminate secrets from code and configuration files
- **Environment Consistency**: Same secret management across dev, staging, and
  production
- **Easier Rotation**: Automated secret rotation without service restarts
- **Better Developer Experience**: Secure local development without exposing
  real secrets

## Migration Process

### Prerequisites

1. **Vault Server**: Ensure Vault is running and accessible

   ```bash
   docker compose -f docker-compose.dev.yml up -d vault
   curl http://localhost:8200/v1/sys/health
   ```

2. **Initial Setup**: Configure Vault with policies and secrets
   ```bash
   ./scripts/vault-setup.sh full
   ```

### Migration Steps

#### 1. Backup Current Configuration

```bash
# Create backups before making changes
./scripts/migrate-to-vault.sh backup
```

#### 2. Run Migration Script

```bash
# Complete migration
./scripts/migrate-to-vault.sh migrate
```

#### 3. Validate Migration

```bash
# Test that everything works
./scripts/migrate-to-vault.sh validate
```

## What the Migration Does

### 1. Updates Docker Compose Configuration

**Before:**

```yaml
services:
  ai:
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - JWT_SECRET=${JWT_SECRET}
```

**After:**

```yaml
services:
  ai:
    environment:
      - VAULT_ADDR=http://vault:8200
      - SERVICE_NAME=ai
      - VAULT_NAMESPACE=lilith
```

### 2. Creates Vault Client Library

**File:** `services/lib/vault-client-enhanced.js`

Features:

- Automatic token management
- Caching for performance
- Encryption/decryption support
- Development and production modes
- Error handling and retry logic

### 3. Updates Service Configuration

Each service gets:

- **Vault Configuration File**: `.env.vault`
- **Integration Module**: `src/vault-config.js`
- **Startup Updates**: Load config from Vault

### 4. Service-Specific Integrations

#### AI Service

```javascript
import { loadAIConfig } from './vault-config.js';

const config = await loadAIConfig();
// Now has: config.openai_api_key, config.anthropic_api_key, etc.
```

#### Auth Service

```javascript
import { loadAuthConfig } from './vault-config.js';

const config = await loadAuthConfig();
// Now has: config.jwt_secret, config.password_salt, etc.
```

#### Notification Service

```javascript
import { loadNotificationConfig } from './vault-config.js';

const config = await loadNotificationConfig();
// Now has: config.sendgrid_api_key, config.twilio_auth_token, etc.
```

## Post-Migration Validation

### 1. Health Check

```bash
# Check Vault is accessible
curl http://localhost:8200/v1/sys/health

# Validate migration
./scripts/validate-vault-migration.sh
```

### 2. Service Testing

```bash
# Start services
docker compose -f docker-compose.dev.yml up -d

# Check service logs
docker logs lilith-ai-1
docker logs lilith-auth-1
```

### 3. Secret Access Test

```bash
# Test Vault access from services
docker exec lilith-ai-1 node -e "
const { createVaultClient } = require('./services/lib/vault-client-enhanced.js');
const vault = createVaultClient({ serviceName: 'ai' });
vault.getSecret('secret/lilith/ai/config').then(console.log);
"
```

## Service Configuration Details

### Vault Client Configuration

```javascript
const vault = createVaultClient({
  serviceName: process.env.SERVICE_NAME, // e.g., 'ai', 'auth'
  environment: process.env.ENVIRONMENT, // 'development', 'production'
  addr: process.env.VAULT_ADDR, // 'http://vault:8200'
  token: process.env.VAULT_TOKEN, // Auto-loaded from file/env
  secretPath: 'secret/lilith/ai', // Service-specific path
  sharedPath: 'secret/lilith/shared', // Shared secrets path
  cacheTimeout: 300000, // 5 minutes
});
```

### Secret Structure in Vault

```
secret/lilith/
├── shared/
│   ├── database          # Database credentials
│   ├── jwt              # JWT secrets
│   ├── storage          # S3/MinIO keys
│   ├── redis            # Redis configuration
│   └── external-apis    # Third-party API keys
├── ai/
│   └── config           # AI service secrets
├── auth/
│   └── config           # Auth service secrets
├── notification/
│   └── config           # Notification service secrets
└── media/
    └── config           # Media service secrets
```

### Access Patterns

#### Shared Secrets

```javascript
// Any service can access shared secrets
const dbConfig = await vault.getSharedSecret('database');
const jwtConfig = await vault.getSharedSecret('jwt');
```

#### Service-Specific Secrets

```javascript
// Only the AI service can access its secrets
const aiConfig = await vault.getServiceSecret('config');
```

#### Direct Path Access

```javascript
// Direct access to any secret
const secret = await vault.getSecret('secret/lilith/shared/database');
```

## Security Considerations

### 1. Token Management

- **Service Tokens**: Long-lived tokens for services
- **Human Tokens**: Short-lived tokens for developers
- **Auto-Renewal**: Automatic token renewal before expiration

### 2. Access Control

- **Least Privilege**: Services only access secrets they need
- **Policy Enforcement**: Vault policies prevent unauthorized access
- **Audit Logging**: All access logged and monitored

### 3. Secret Rotation

- **Automated Rotation**: Configure automatic secret rotation
- **Versioning**: Keep multiple versions of secrets
- **Rollback**: Ability to rollback to previous secret versions

## Development Workflow

### 1. Local Development

```bash
# Start Vault in development mode
docker compose -f docker-compose.dev.yml up -d vault

# Setup initial secrets
./scripts/vault-setup.sh secrets

# Work with services normally
docker compose -f docker-compose.dev.yml up -d ai
```

### 2. Adding New Secrets

**Via Vault UI:**

1. Open http://localhost:8200
2. Navigate to `secret/`
3. Add new secret under appropriate path

**Via CLI:**

```bash
# Add shared secret
vault kv put secret/lilith/shared/new-secret \
  api_key="your-api-key" \
  endpoint="https://api.example.com"

# Add service-specific secret
vault kv put secret/lilith/ai/new-model \
  model_name="gpt-4-turbo" \
  api_key="your-model-key"
```

**Via Script:**

```bash
# Use setup script
./scripts/vault-setup.sh manage
```

### 3. Testing New Secrets

```javascript
// In service code
const newSecret = await vault.getSharedSecret('new-secret');
console.log(newSecret.api_key);
```

## Production Considerations

### 1. Vault Configuration

```yaml
# docker-compose.prod.yml
vault:
  environment:
    - VAULT_MODE=server
    - VAULT_ADDR=https://vault.example.com
    - VAULT_TLS_CERT_FILE=/vault/tls/cert.pem
    - VAULT_TLS_KEY_FILE=/vault/tls/key.pem
  volumes:
    - ./vault/config:/vault/config
    - vault-data:/vault/data
    - ./vault/tls:/vault/tls
```

### 2. Service Configuration

```bash
# Production environment variables
export VAULT_ADDR=https://vault.example.com
export VAULT_NAMESPACE=lilith
export VAULT_TLS_SKIP_VERIFY=false
```

### 3. Authentication Methods

**Kubernetes Auth:**

```javascript
// Service authenticates with Kubernetes service account
const vault = createVaultClient({
  authMethod: 'kubernetes',
  roleName: 'ai-service',
});

await vault.authenticateK8s();
```

**AWS IAM Auth:**

```javascript
// Service authenticates with AWS IAM
const vault = createVaultClient({
  authMethod: 'aws-iam',
  role: 'ai-service',
});
```

## Troubleshooting

### Common Issues

#### 1. Service Cannot Connect to Vault

```bash
# Check Vault connectivity
docker exec lilith-ai-1 curl -f http://vault:8200/v1/sys/health

# Check network
docker network ls | grep lilith
```

#### 2. Token Authentication Failed

```bash
# Check token validity
vault token lookup $VAULT_TOKEN

# Create new service token
./scripts/vault-setup.sh accounts
```

#### 3. Secret Not Found

```bash
# Check if secret exists
vault kv get secret/lilith/shared/database

# List available secrets
vault kv list secret/lilith/
```

#### 4. Permission Denied

```bash
# Check service policies
vault policy read ai

# Verify token permissions
vault token lookup -accessor=$TOKEN_ACCESSOR
```

### Debug Mode

Enable debug logging:

```bash
# Set environment variable
export VAULT_LOG_LEVEL=debug

# Or update service configuration
export DEBUG=vault:*
```

### Cache Issues

Clear Vault client cache:

```javascript
// In service code
vault.clearCache();
```

## Best Practices

### 1. Development

- **Never commit secrets** to version control
- **Use development tokens** for local development
- **Test secret rotation** procedures regularly
- **Validate access** patterns in staging

### 2. Production

- **Use TLS** for all Vault communication
- **Enable audit logging** and monitor regularly
- **Implement automatic token renewal**
- **Regular security audits** of Vault policies

### 3. Operations

- **Backup Vault data** regularly
- **Test disaster recovery** procedures
- **Monitor Vault performance** and health
- **Document all procedures** and configurations

## Rollback Plan

If migration fails or causes issues:

### 1. Stop Services

```bash
docker compose -f docker-compose.dev.yml down
```

### 2. Restore Backups

```bash
# Find latest backup
ls -la backups/vault-migration-*

# Restore docker-compose.dev.yml
cp backups/vault-migration-*/docker-compose.dev.yml .
```

### 3. Restart Services

```bash
docker compose -f docker-compose.dev.yml up -d
```

### 4. Validate

```bash
# Check services are working with old config
docker compose -f docker-compose.dev.yml ps
```

## Monitoring and Maintenance

### 1. Vault Health Monitoring

```bash
# Health check script
./scripts/health-monitor.sh utilities

# Vault-specific metrics
curl -s http://localhost:8200/v1/sys/metrics | jq
```

### 2. Secret Usage Monitoring

```bash
# Check token usage
vault token lookup -accessor=$TOKEN_ACCESSOR

# Audit log monitoring
tail -f vault/logs/audit.log | jq '.request.path'
```

### 3. Performance Monitoring

```bash
# Monitor response times
curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8200/v1/sys/health

# Monitor cache hit rates
# (Add metrics to your vault client)
```

## Next Steps

After migration:

1. **Test Thoroughly**: Ensure all services work with Vault
2. **Monitor Performance**: Check for any performance impact
3. **Train Team**: Ensure team members understand Vault usage
4. **Update Documentation**: Keep all documentation current
5. **Regular Maintenance**: Schedule regular secret rotation and cleanup

## Support

- **Documentation**: See `docs/deploy/VAULT_SETUP.md`
- **Scripts**: See `scripts/vault-setup.sh`
- **Health Monitoring**: See `scripts/health-monitor.sh`
- **Issues**: Create GitHub issues for problems
