Eve library

Lilith Operations Guide

Guide

Revision 1

On this page

Overview#

This guide covers how to run, monitor, debug, and maintain the Lilith platform in production and development environments.

Table of Contents#

  1. Local Development
  2. Deployment
  3. Monitoring
  4. Alerting
  5. Incident Response
  6. Database Operations
  7. Troubleshooting
  8. Runbook Quick Reference

Local Development#

Prerequisites#

  • Node.js 18+
  • Docker and Docker Compose
  • PostgreSQL 15+
  • Redis 7+

Quick Start#

bash
# Clone and install
git clone https://github.com/lilith/lilith
cd lilith
npm install

# Start infrastructure
docker compose up -d postgres redis kafka milvus

# Seed development data
npm run seed:dev

# Start services
npm run deploy

# Verify health
npm run health:wait

Service Ports#

Service Port Description
BFF 8000 API Gateway
Auth 8001 Authentication
Content 8002 Content Management
Conversation 8003 Chat/Messages
AI 8004 AI Orchestration
Media 8005 Media/Assets
TTS 8011 Text-to-Speech
STT 8010 Speech-to-Text

Development Commands#

bash
# View logs
npm run logs              # All services
npm run logs:errors       # Errors only
npm run logs -- --service ai  # Specific service

# Run tests
npm test                  # All tests
npm run test:e2e          # E2E tests
npm run test:coverage     # With coverage

# Code quality
npm run lint              # Lint check
npm run format            # Format code
npm run typecheck         # Type check

Deployment#

Environments#

Environment URL Purpose
Local localhost:8000 Development
Staging staging.api.lilith.app Pre-production
Production api.lilith.app Live users

Kubernetes Deployment#

bash
# Deploy to staging
kubectl apply -k deploy/kubernetes/overlays/staging

# Deploy to production
kubectl apply -k deploy/kubernetes/overlays/production

# Check rollout status
kubectl rollout status deployment/ai-service -n lilith

Helm Deployment#

bash
# Install/upgrade
helm upgrade --install lilith deploy/helm/lilith-service \
  --namespace lilith \
  --values deploy/helm/values-production.yaml

# Rollback
helm rollback lilith 1 -n lilith

ArgoCD (GitOps)#

bash
# Sync application
argocd app sync lilith-app-of-apps

# Check status
argocd app get lilith-app-of-apps

# Manual sync (emergency)
argocd app sync lilith-app-of-apps --force

Rolling Updates#

  1. Update image tag in values file
  2. Commit to main branch
  3. ArgoCD auto-syncs (or manual sync)
  4. Monitor rollout health
  5. Rollback if issues detected

Monitoring#

Key Dashboards#

Dashboard URL Purpose
Service Health /grafana/d/services Overall system health
AI Metrics /grafana/d/ai LLM latency, tokens
TTS/STT /grafana/d/speech Audio service metrics
User Activity /grafana/d/users DAU, sessions

Key Metrics#

Latency (SLO Targets)#

Metric p50 Target p95 Target
Chat Response < 2s < 4s
TTS First Byte < 600ms < 1s
API Endpoints < 200ms < 500ms

Error Rates#

Metric Target
Error Rate < 0.1%
5xx Rate < 0.01%
Circuit Breaker Trips < 5/hour

Health Checks#

bash
# Service health endpoint
curl https://api.lilith.app/v1/health

# Kubernetes pod health
kubectl get pods -n lilith -o wide

# Service mesh health
istioctl proxy-status

Logging#

Log Aggregation#

Logs flow to ELK stack:

text
Service → Fluentd → Elasticsearch → Kibana

Log Queries#

bash
# Kibana: Find errors for AI service
service:ai AND level:error AND @timestamp:[now-1h TO now]

# Find slow requests
response_time_ms:>2000 AND service:bff

# Trace specific request
correlation_id:"abc123"

Tracing#

Jaeger UI#

Access at: /jaeger

Trace a request:

  1. Get correlation ID from logs or response header
  2. Search in Jaeger by trace ID
  3. View span waterfall

Alerting#

Alert Channels#

Severity Channel Response Time
Critical PagerDuty < 15 min
High Slack #alerts < 1 hour
Medium Email < 4 hours
Low Ticket Next business day

Critical Alerts#

Alert Condition Action
Service Down Pod restarts > 3 in 5min Check logs, restart
High Error Rate 5xx > 1% for 5min Investigate, rollback
Database Unreachable Connection failures Check DB, failover
TTS Latency p95 > 2s for 10min Scale up, check GPU

Alert Response#

  1. Acknowledge alert in PagerDuty
  2. Check dashboard for scope
  3. Review recent deployments
  4. Check logs for root cause
  5. Apply mitigation
  6. Update incident channel
  7. Post-incident review

Incident Response#

Severity Levels#

Level Definition Response
SEV-1 Complete outage All hands, < 15min
SEV-2 Major feature down On-call + backup, < 30min
SEV-3 Degraded performance On-call, < 1hr
SEV-4 Minor issue Business hours

Incident Process#

  1. Detect: Alert fires or user report
  2. Triage: Assess severity, notify stakeholders
  3. Mitigate: Apply quick fix (rollback, scale, restart)
  4. Communicate: Status page, Slack updates
  5. Resolve: Fix root cause
  6. Review: Post-incident analysis

Communication Templates#

Initial:

We are investigating reports of [issue]. Updates to follow.

Mitigation:

We have identified the issue as [cause] and are applying [fix]. Service should recover within [time].

Resolution:

The issue has been resolved. [Summary]. We will conduct a post-incident review.


Database Operations#

PostgreSQL#

Connection#

bash
# Production (via bastion)
psql -h db.lilith.internal -U lilith -d lilith

# Local
psql -h localhost -U postgres -d lilith

Common Operations#

sql
-- Check connections
SELECT count(*) FROM pg_stat_activity;

-- Slow queries
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC LIMIT 10;

-- Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

Backup/Restore#

bash
# Backup
pg_dump -h db.lilith.internal -U lilith lilith > backup.sql

# Restore
psql -h db.lilith.internal -U lilith lilith < backup.sql

Redis#

bash
# Connect
redis-cli -h redis.lilith.internal

# Memory usage
INFO memory

# Key patterns
KEYS "session:*" | wc -l

Milvus (Vector DB)#

bash
# Health check
curl http://milvus.lilith.internal:19530/v1/health

# Collection stats
curl http://milvus.lilith.internal:19530/v1/collections/content_embeddings

Troubleshooting#

Common Issues#

High Latency#

  1. Check service metrics in Grafana
  2. Review traces in Jaeger
  3. Check database query times
  4. Verify network latency (service mesh)
  5. Check for resource contention
bash
# Check pod resources
kubectl top pods -n lilith

# Check network
istioctl proxy-status

Pod CrashLoopBackOff#

  1. Check pod logs
  2. Verify environment variables
  3. Check resource limits
  4. Verify health check endpoints
  5. Check dependencies (DB, cache)
bash
kubectl logs -n lilith <pod-name> --previous
kubectl describe pod -n lilith <pod-name>

Database Connection Issues#

  1. Check connection pool usage
  2. Verify credentials
  3. Check network policies
  4. Test connectivity from pod
bash
# Test from pod
kubectl exec -it <pod> -n lilith -- nc -zv postgres 5432

TTS/STT Failures#

  1. Check GPU availability
  2. Verify model loading
  3. Check queue depth
  4. Review error logs
bash
# GPU status
nvidia-smi

# TTS queue
curl http://tts:8011/v1/queue/status

Debug Mode#

bash
# Enable debug logging (temporary)
kubectl set env deployment/ai-service LOG_LEVEL=debug -n lilith

# Revert
kubectl set env deployment/ai-service LOG_LEVEL=info -n lilith

Runbook Quick Reference#

Emergency Procedures#

Scenario Command/Action
Service unresponsive kubectl rollout restart deployment/<svc> -n lilith
Database overload Scale read replicas, kill long queries
High error rate Rollback: helm rollback lilith
DDoS detected Enable WAF rules, scale up
Secrets compromised Rotate keys via Vault

Scaling#

bash
# Manual scale
kubectl scale deployment/ai-service --replicas=10 -n lilith

# HPA adjustment (temporary)
kubectl patch hpa ai-service -n lilith -p '{"spec":{"maxReplicas":50}}'

Maintenance Windows#

Task Frequency Impact
Database backup Daily None
Index rebuild Weekly Minor latency
Certificate renewal Quarterly Rolling update
Dependency updates Monthly Scheduled downtime