# Lilith Local Development Environment Guide

## Overview

This guide provides comprehensive instructions for setting up and running the
Lilith platform locally using both Docker Compose and Kind Kubernetes cluster
configurations. The platform supports two development environments:

1. **Docker Compose Development** - Fast startup, lightweight, ideal for
   individual development
2. **Kind Kubernetes Development** - Production-like environment, supports
   advanced Kubernetes features

## Prerequisites

### System Requirements

- **Operating System**: macOS, Linux, or Windows (with WSL2)
- **Memory**: Minimum 16GB RAM (32GB recommended for full stack)
- **Storage**: 50GB free disk space
- **CPU**: 4+ cores (8+ recommended)

### Required Tools

#### Essential Tools

- **Docker Desktop** (or Docker Engine + Docker Compose)
- **Node.js** (v18+ recommended)
- **npm** or **yarn**
- **Git**

#### For Kubernetes Development

- **Kind** (Kubernetes in Docker)
- **kubectl** (Kubernetes CLI)
- **Helm** (optional, for advanced observability stack)

#### Development Tools (Recommended)

- **VS Code** with Docker and Kubernetes extensions
- **Postman** or **Insomnia** for API testing
- **DBeaver** or **DataGrip** for database access

## Quick Start

### Option 1: Docker Compose (Recommended for most developers)

```bash
# Clone the repository
git clone <repository-url>
cd lilith

# Complete setup (installs dependencies and starts services)
make quick-start

# Or do it step by step:
make setup    # Install dependencies
make dev      # Start all services
make seed     # Seed development data
```

### Option 2: Kind Kubernetes (Advanced users)

```bash
# Complete Kubernetes setup
make quick-start-k8s

# Or do it step by step:
make setup           # Install dependencies
make kind-setup      # Create Kind cluster
make kind-db         # Deploy databases
make kind-deploy     # Deploy services
```

### Minimal service-only workflow (no containers)

For quick iteration on a single service:

- Install deps once: `npm ci`
- Work inside `services/<service>` and run targeted tests:
  `npm test --workspace services/<service>`
- Prefer Fastify `app.inject()` in tests; avoid `app.listen()` during local
  checks
- Validate contracts: `npm run openapi:validate`
- Keep coverage ≥70%: `npm run coverage:ci`

## Architecture Overview

### Docker Compose Environment

The Docker Compose setup includes:

- **50+ Microservices**: All Lilith platform services
- **6 Database Services**: PostgreSQL, Milvus, Redis, Elasticsearch, Neo4j,
  MinIO
- **Development Tools**: Prometheus, Grafana, Jaeger, Vault, nginx-proxy
- **Port Mapping**: Services accessible on ports 3000-3100
- **Live Reload**: Development volumes mounted for hot reloading

### Kind Kubernetes Environment

The Kind setup provides:

- **3-Node Cluster**: 1 control plane + 2 worker nodes
- **Namespace Isolation**: Services organized by function
- **Production-like Networking**: Calico CNI, Ingress controller
- **Observability Stack**: Prometheus, Grafana, Jaeger integration
- **Security**: Namespace policies, resource quotas
- **Port Compatibility**: Same port mappings as Docker Compose

## Service Architecture

### Core Services

| Service          | Port | Description                      | Dependencies              |
| ---------------- | ---- | -------------------------------- | ------------------------- |
| **BFF**          | 3000 | Backend-for-Frontend API Gateway | All services              |
| **Auth**         | 3003 | Authentication & JWT Management  | PostgreSQL, Redis         |
| **AI**           | 3002 | LLM Orchestration with RAG       | Milvus, Redis, Moderation |
| **Conversation** | 3001 | Real-time Chat & Threading       | PostgreSQL, Redis         |
| **Content**      | 3004 | Content Management & Versioning  | PostgreSQL, Elasticsearch |
| **Media**        | 3005 | Media Processing & Storage       | PostgreSQL, MinIO         |
| **Notification** | 3006 | Push/Email Dispatch              | PostgreSQL                |
| **Moderation**   | 3007 | Content Safety & Policy          | PostgreSQL, Redis         |
| **RAG**          | 3008 | Retrieval-Augmented Generation   | Milvus, Elasticsearch     |
| **TTS**          | 3009 | Text-to-Speech Synthesis         | GPU (optional)            |
| **STT**          | 3010 | Speech-to-Text Recognition       | GPU (optional)            |

### Database Services

| Database          | Port      | Purpose                        | Access Information                             |
| ----------------- | --------- | ------------------------------ | ---------------------------------------------- |
| **PostgreSQL**    | 5433      | Primary relational database    | User: `lilith`, Password: `dev_password`       |
| **Milvus**        | 19530     | Vector database for embeddings | Web UI: http://localhost:9001                  |
| **Redis**         | 6380      | Caching & session storage      | No authentication (dev)                        |
| **Elasticsearch** | 9201      | Full-text search               | No authentication (dev)                        |
| **Neo4j**         | 7475/7688 | Knowledge graph database       | User: `neo4j`, Password: `dev_password`        |
| **MinIO**         | 9000/9001 | S3-compatible object storage   | Access Key: `minioadmin`, Secret: `minioadmin` |

### Development Tools

| Tool                | Port  | Description                | Credentials            |
| ------------------- | ----- | -------------------------- | ---------------------- |
| **Prometheus**      | 9090  | Metrics collection         | No auth (dev)          |
| **Grafana**         | 9092  | Visualization & dashboards | admin/admin            |
| **Jaeger**          | 16686 | Distributed tracing        | No auth (dev)          |
| **Vault**           | 8200  | Secrets management         | Token: `dev-token`     |
| **nginx-proxy**     | 8081  | Reverse proxy              | No auth (dev)          |
| **Redis Commander** | 8082  | Redis GUI                  | No auth (dev)          |
| **pgAdmin**         | 8083  | PostgreSQL GUI             | admin@lilith.dev/admin |

## Development Workflow

### 1. Initial Setup

```bash
# Install all dependencies for all services
make setup

# Create environment file (if not exists)
cp .env.dev.example .env.dev
# Edit .env.dev with your configuration
```

### 2. Starting Services

#### Docker Compose

```bash
# Start all services in foreground
make dev

# Start in background
make dev-detached

# Check service health
make health

# View logs
make logs
```

#### Kind Kubernetes

```bash
# Setup cluster and databases
make kind-setup kind-db

# Deploy services
make kind-deploy

# Check cluster status
make kind-status

# View specific service logs
make kind-logs NAMESPACE=lilith POD=bff
```

### 3. Development Tasks

#### Service Development

```bash
# View logs for specific service
make logs-service SERVICE=bff

# Restart specific service
make restart-service SERVICE=bff

# Open shell in service container
make dev-shell  # Docker Compose
make kind-shell NAMESPACE=lilith POD=bff  # Kind
```

#### Database Operations

```bash
# Reset all databases
make db-reset

# Backup development data
make backup-data

# Reset and re-seed data
make seed-reset
```

#### Testing

```bash
# Run all tests
make test

# Run tests for specific service
make test-service SERVICE=conversation

# Run performance tests
make perf-test

# Run load test
make load-test
```

## API Access

### Local Endpoints

#### Primary API Gateway (BFF)

- **Base URL**: `http://localhost:3000`
- **Health**: `GET http://localhost:3000/v1/health`
- **Chat**: `POST http://localhost:3000/v1/chat`
- **Dashboard**: `GET http://localhost:3000/v1/dashboard?user_id=USER_ID`

#### Direct Service Access

- **Conversation**: `http://localhost:3001`
- **AI**: `http://localhost:3002`
- **Auth**: `http://localhost:3003`
- **Content**: `http://localhost:3004`
- **Media**: `http://localhost:3005`
- **Notification**: `http://localhost:3006`
- **Moderation**: `http://localhost:3007`
- **RAG**: `http://localhost:3008`
- **TTS**: `http://localhost:3009`
- **STT**: `http://localhost:3010`

### API Authentication

1. **Register a user**:

   ```bash
   curl -X POST http://localhost:3003/v1/register \
     -H "Content-Type: application/json" \
     -d '{"email":"dev@example.com","password":"dev123456"}'
   ```

2. **Login to get token**:

   ```bash
   curl -X POST http://localhost:3003/v1/login \
     -H "Content-Type: application/json" \
     -d '{"email":"dev@example.com","password":"dev123456"}'
   ```

3. **Use token in API calls**:
   ```bash
   curl -X POST http://localhost:3000/v1/chat \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer YOUR_JWT_TOKEN" \
     -d '{"persona":"zen_master","input":{"text":"Hello"}}'
   ```

## Development Best Practices

### 1. Code Organization

- Follow the existing service structure in `services/`
- Use TypeScript with strict mode enabled
- Implement proper error handling and logging
- Write tests for new functionality

### 2. Environment Management

- Use `.env.dev` for local configuration
- Never commit secrets to version control
- Use Vault for sensitive configuration in production

### 3. Database Development

- Use migrations for schema changes
- Seed data for development and testing
- Use transaction rollbacks in tests

### 4. API Development

- Follow RESTful conventions
- Implement proper HTTP status codes
- Use OpenAPI specifications
- Validate input data

### 5. Testing

- Write unit tests for business logic
- Write integration tests for API endpoints
- Use E2E tests for critical user journeys
- Mock external dependencies in tests

## Troubleshooting

### Common Issues

#### Port Conflicts

```bash
# Check what's using ports
lsof -i :3000-3010

# Kill processes on specific ports
kill -9 $(lsof -ti:3000)
```

#### Docker Issues

```bash
# Check Docker status
docker info

# Reset Docker state
docker system prune -a
docker volume prune
```

#### Memory Issues

```bash
# Check memory usage
docker stats

# Increase Docker memory allocation in Docker Desktop settings
```

#### Service Startup Issues

```bash
# Check service logs
make logs-service SERVICE=bff

# Restart specific service
make restart-service SERVICE=bff

# Reset and restart everything
make full-reset
```

### Health Checks

```bash
# Check all service health
make health

# Check Kubernetes cluster health
make kind-status

# Verify database connections
docker compose -f docker/docker-compose.dev.yml exec postgres pg_isready -U lilith
docker compose -f docker/docker-compose.dev.yml exec redis redis-cli ping
```

### Debugging Tips

1. **Use container shells**:

   ```bash
   make dev-shell  # Docker Compose
   make kind-shell NAMESPACE=lilith POD=bff  # Kind
   ```

2. **Check service dependencies**:

   ```bash
   # View dependency graph
   docker compose -f docker/docker-compose.dev.yml config --services
   ```

3. **Monitor resource usage**:

   ```bash
   # Docker stats
   docker stats

   # Kubernetes resource usage
   kubectl top pods --all-namespaces
   ```

## Performance Optimization

### Development Performance

1. **Use volume mounts**: Enabled by default for hot reloading
2. **Limit service scope**: Start only services you need
3. **Use development images**: Optimized for faster startup
4. **Cache dependencies**: Docker layer caching enabled

### Resource Management

```bash
# Limit resource usage in Docker Compose
# Edit docker-compose.dev.yml to add resource limits

# Monitor Kubernetes resources
kubectl describe nodes
kubectl get resourcequotas -n lilith
```

## Environment Switching

### Switch Between Docker and Kubernetes

```bash
# From Docker to Kubernetes
make switch-to-k8s

# From Kubernetes to Docker
make switch-to-docker
```

### Clean Environment Reset

```bash
# Docker Compose reset
make full-reset

# Kubernetes reset
make full-reset-k8s
```

## Contributing

### Setting Up for Contributions

1. Fork the repository
2. Create a feature branch
3. Set up local development environment
4. Make your changes
5. Run tests and ensure everything works
6. Submit a pull request

### Pre-commit Hooks

The project includes pre-commit hooks for:

- TypeScript compilation
- Linting and formatting
- Test execution
- Security scanning

### CI/CD Integration

- GitHub Actions run tests on every PR
- Automated E2E tests validate functionality
- Performance tests ensure quality standards

## Additional Resources

### Documentation

- **API Documentation**: `http://localhost:3000/docs` (when running)
- **Architecture Guide**: `ARCHITECTURE.md`
- **Product Specifications**: `PRODUCT.md`
- **Security Guidelines**: `docs/security/`

### Monitoring and Observability

- **Grafana Dashboards**: `http://localhost:9092` (admin/admin)
- **Prometheus Metrics**: `http://localhost:9090`
- **Jaeger Tracing**: `http://localhost:16686`
- **Service Logs**: `make logs` or `make kind-logs`

### Community and Support

- **Issues**: GitHub repository issues
- **Discussions**: GitHub repository discussions
- **Documentation**: Project documentation site

---

## Quick Reference

### Essential Commands

```bash
# Docker Compose
make setup          # Initial setup
make dev            # Start all services
make health         # Check service health
make logs           # View all logs
make down           # Stop services
make clean          # Clean up everything

# Kind Kubernetes
make kind-setup     # Setup cluster
make kind-deploy    # Deploy services
make kind-status    # Check cluster status
make kind-down      # Destroy cluster

# Testing
make test           # Run all tests
make test-service SERVICE=bff    # Test specific service
make perf-test      # Performance tests

# Utilities
make api-docs       # Show API endpoints
make backup-data    # Backup development data
make full-reset     # Complete environment reset
```

### Access URLs

| Service        | URL                    | Description         |
| -------------- | ---------------------- | ------------------- |
| **Main API**   | http://localhost:3000  | BFF API Gateway     |
| **Grafana**    | http://localhost:9092  | Metrics dashboard   |
| **Prometheus** | http://localhost:9090  | Metrics collection  |
| **Jaeger**     | http://localhost:16686 | Distributed tracing |
| **Vault**      | http://localhost:8200  | Secrets management  |
| **pgAdmin**    | http://localhost:8083  | PostgreSQL GUI      |
| **MinIO**      | http://localhost:9001  | Object storage GUI  |

For any issues or questions, please refer to the project documentation or create
an issue in the repository.
