# HashiCorp Vault Setup and Configuration

## Overview

This document describes the HashiCorp Vault implementation for centralized
secret management in the Lilith platform. Vault provides secure storage, dynamic
secrets, encryption as a service, and fine-grained access control.

## Architecture

### Components

1. **Vault Server**: Core secret management service
2. **Secrets Engines**: KV v2, Transit, PKI
3. **Policies**: Role-based access control
4. **Authentication**: Token-based auth for services
5. **Audit Logging**: Comprehensive access logging

### Directory Structure

```
vault/
├── config.hcl                 # Vault configuration
├── policies/
│   └── policies.hcl           # Access control policies
├── scripts/
│   └── setup.sh               # Container initialization
├── docker-entrypoint.sh       # Custom entrypoint
├── tokens/                    # Service tokens
├── backups/                   # Backup storage
└── logs/                      # Audit logs
```

## Quick Start

### 1. Start Vault

```bash
# Start with development environment
docker compose -f docker-compose.dev.yml up -d vault

# Check Vault status
curl http://localhost:8200/v1/sys/health
```

### 2. Initialize Vault

```bash
# Run complete setup
./scripts/vault-setup.sh full

# Or step by step
./scripts/vault-setup.sh init
./scripts/vault-setup.sh secrets
./scripts/vault-setup.sh policies
./scripts/vault-setup.sh accounts
```

### 3. Access Vault UI

- **URL**: http://localhost:8200
- **Token**: dev-token
- **Method**: Token

## Configuration

### Vault Configuration (config.hcl)

Key configuration options:

```hcl
# UI and API
ui = true
api_addr = "http://0.0.0.0:8200"

# Backend
backend "file" {
  path = "/vault/data"
}

# Listener
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

# Audit logging
audit "file" {
  path    = "/vault/logs/audit.log"
  format  = "json"
}

# Lease TTLs
default_lease_ttl = "168h"
max_lease_ttl = "720h"
```

### Environment Variables

| Variable                  | Default                 | Description                |
| ------------------------- | ----------------------- | -------------------------- |
| `VAULT_ADDR`              | `http://localhost:8200` | Vault server URL           |
| `VAULT_TOKEN`             | `dev-token`             | Root token for development |
| `VAULT_MODE`              | `dev`                   | Server mode (dev/server)   |
| `VAULT_DEV_ROOT_TOKEN_ID` | `dev-token`             | Development root token     |

## Secrets Management

### Secrets Engines

#### 1. KV v2 (Key-Value)

General purpose secret storage:

```bash
# Store a secret
vault kv put secret/lilith/ai/config \
  openai_api_key="sk-..." \
  anthropic_api_key="sk-ant-..."

# Read a secret
vault kv get secret/lilith/ai/config

# List secrets
vault kv list secret/lilith/
```

#### 2. Transit

Encryption as a service:

```bash
# Create encryption key
vault write -f transit/keys/data

# Encrypt data
vault write transit/encrypt/data \
  plaintext=$(base64 <<< "sensitive data")

# Decrypt data
vault write transit/decrypt/data \
  ciphertext="vault:v1:..."
```

#### 3. PKI

Certificate management:

```bash
# Generate certificate
vault write pki/issue/lilith-server \
  common_name="bff.lilith.local" \
  ttl="24h"
```

### Secret Structure

```
secret/lilith/
├── shared/
│   ├── database         # Database credentials
│   ├── jwt             # JWT secrets
│   ├── storage         # S3/MinIO keys
│   ├── redis           # Redis configuration
│   ├── vector-db       # Milvus configuration
│   └── external-apis   # Third-party API keys
├── bff/                # BFF-specific secrets
├── ai/                 # AI service secrets
├── auth/               # Auth service secrets
├── content/            # Content service secrets
├── media/              # Media service secrets
├── notification/       # Notification service secrets
├── moderation/         # Moderation service secrets
├── rag/                # RAG service secrets
├── tts/                # TTS service secrets
└── stt/                # STT service secrets
```

## Access Control

### Policies

Vault policies define what each service can access:

#### Example Service Policy (BFF)

```hcl
# Read service-specific secrets
path "secret/lilith/bff/*" {
  capabilities = ["read"]
}

# Read shared secrets
path "secret/lilith/shared/jwt" {
  capabilities = ["read"]
}

path "secret/lilith/shared/database" {
  capabilities = ["read"]
}
```

#### Service Accounts

Each service gets a dedicated token with appropriate permissions:

```bash
# Create service token
vault token create -policy=bff -ttl=720h

# Service uses token to access secrets
export VAULT_TOKEN="service-token"
vault kv get secret/lilith/bff/config
```

### Policy Hierarchy

1. **admin**: Full access to everything
2. **operator**: System management, limited secret access
3. **service policies**: Service-specific secret access
4. **readonly**: Read-only access to shared secrets

## Integration with Services

### Service Configuration

Services are configured to use Vault through environment variables and
configuration files:

```bash
# .env.vault for each service
VAULT_ADDR=http://vault:8200
VAULT_TOKEN_FILE=/vault/tokens/bff.token
VAULT_SECRET_PATH=secret/lilith/bff
VAULT_SHARED_PATH=secret/lilith/shared
```

### Node.js Integration Example

```javascript
// vault-client.js
const axios = require('axios');

class VaultClient {
  constructor(opts = {}) {
    this.addr = opts.addr || process.env.VAULT_ADDR;
    this.token = opts.token || process.env.VAULT_TOKEN;
    this.secretPath = opts.secretPath || process.env.VAULT_SECRET_PATH;
  }

  async getSecret(path) {
    const response = await axios.get(
      `${this.addr}/v1/${this.secretPath}/${path}`,
      {
        headers: { 'X-Vault-Token': this.token },
      }
    );
    return response.data.data.data;
  }

  async getSharedSecret(path) {
    const response = await axios.get(
      `${this.addr}/v1/secret/lilith/shared/${path}`,
      {
        headers: { 'X-Vault-Token': this.token },
      }
    );
    return response.data.data.data;
  }
}

module.exports = VaultClient;
```

### Service Usage

```javascript
// In service startup
const VaultClient = require('./vault-client');

const vault = new VaultClient();

async function loadConfig() {
  try {
    const dbConfig = await vault.getSharedSecret('database');
    const jwtConfig = await vault.getSharedSecret('jwt');

    return {
      database: dbConfig,
      jwt: jwtConfig,
    };
  } catch (error) {
    console.error('Failed to load config from Vault:', error);
    throw error;
  }
}
```

## Security Features

### Encryption

1. **Data at Rest**: Encrypted storage backend
2. **Data in Transit**: TLS encryption (in production)
3. **Transit Engine**: Encryption as a service
4. **Key Management**: Automatic key rotation

### Audit Logging

All Vault operations are logged:

```json
{
  "time": "2025-01-06T12:00:00Z",
  "type": "request",
  "operation": "read",
  "path": "secret/data/lilith/shared/database",
  "auth": {
    "client_token": "hvs.CAES...",
    "policies": ["bff"]
  }
}
```

### Token Management

1. **Service Tokens**: Long-lived tokens for services
2. **Time-to-Live (TTL)**: Automatic token expiration
3. **Renewal**: Token renewal capabilities
4. **Revocation**: Immediate token invalidation

## Operations

### Monitoring

```bash
# Check Vault status
./scripts/vault-setup.sh status

# Monitor health
curl -s http://localhost:8200/v1/sys/health | jq

# Check token usage
vault token lookup -accessor=your-token-accessor
```

### Backup and Recovery

```bash
# Create backup
./scripts/vault-backup.sh

# Manual backup
vault operator raft snapshot save backup-$(date +%Y%m%d).snap

# Restore (raft mode)
vault operator raft snapshot restore backup-20250106.snap
```

### Maintenance

```bash
# Renew token
vault token renew -increment=24h

# Rotate encryption key
vault write -f transit/keys/data/rotate

# Update policies
vault policy write bff vault/policies/bff.hcl
```

## Environment-Specific Configuration

### Development

```yaml
# docker-compose.dev.yml
vault:
  environment:
    - VAULT_MODE=dev
    - VAULT_DEV_ROOT_TOKEN_ID=dev-token
  volumes:
    - ./vault:/vault
```

### Production

```yaml
# docker-compose.prod.yml
vault:
  environment:
    - VAULT_MODE=server
    - VAULT_ADDR=https://vault.example.com
  volumes:
    - ./vault/config:/vault/config
    - vault-data:/vault/data
```

## Best Practices

### 1. Security

- Use least-privilege policies
- Regular token rotation
- Enable audit logging
- Use TLS in production
- Monitor access patterns

### 2. Operations

- Regular backups
- Monitor health and performance
- Test disaster recovery
- Document procedures
- Automate where possible

### 3. Development

- Use development mode for local development
- Never commit real secrets to git
- Use environment-specific configurations
- Test secret rotation procedures
- Implement proper error handling

## Troubleshooting

### Common Issues

1. **Vault not accessible**

   ```bash
   # Check if running
   docker ps | grep vault

   # Check logs
   docker logs lilith-vault-1
   ```

2. **Permission denied errors**

   ```bash
   # Check token policies
   vault token lookup $VAULT_TOKEN

   # Verify policy syntax
   vault policy read policy-name
   ```

3. **Secret not found**

   ```bash
   # List available secrets
   vault kv list secret/

   # Check path structure
   vault kv get secret/path/to/secret
   ```

### Health Check

```bash
# Comprehensive health check
./scripts/health-monitor.sh utilities

# Vault-specific health
curl -s http://localhost:8200/v1/sys/health | jq '.'
```

### Debug Mode

```bash
# Enable debug logging
export VAULT_LOG_LEVEL=debug

# Check configuration
vault operator init -status
```

## Migration

### From Environment Variables

1. **Create secrets in Vault**

   ```bash
   vault kv put secret/lilith/ai/config \
     openai_api_key="$OPENAI_API_KEY" \
     anthropic_api_key="$ANTHROPIC_API_KEY"
   ```

2. **Update service configuration**

   ```bash
   # Replace environment variables with Vault client
   # Remove secret values from docker-compose.yml
   ```

3. **Deploy updated services**
   ```bash
   docker compose up -d --force-recreate ai-service
   ```

### Migration Checklist

- [ ] Identify all secret environment variables
- [ ] Create corresponding Vault secrets
- [ ] Update service configurations
- [ ] Implement Vault client in services
- [ ] Test secret retrieval
- [ ] Remove secrets from environment files
- [ ] Update documentation
- [ ] Train development team

## References

- [Vault Documentation](https://developer.hashicorp.com/vault/docs)
- [Vault API Reference](https://developer.hashicorp.com/vault/api-docs)
- [Best Practices Guide](https://developer.hashicorp.com/vault/tutorials/best-practices)
- [Security Model](https://developer.hashicorp.com/vault/docs/security)
