Overview#
This document describes the comprehensive health monitoring and automatic restart system implemented for the Lilith platform. The system provides real-time monitoring, automated recovery, and detailed health reporting for all services.
Architecture#
Components#
- Docker Health Checks: Native Docker health checks for all services
- Restart Policies: Automated service recovery based on health status
- Resource Limits: Memory and CPU constraints to prevent resource exhaustion
- Monitoring Scripts: Custom scripts for comprehensive health analysis
- Alerting: Critical service notifications and error detection
Configuration Files#
docker-compose.health.yml- Health checks and restart policies overlayscripts/health-monitor.sh- Comprehensive monitoring scriptdocker-compose.dev.yml- Base configuration with health checks
Health Check Configuration#
Application Services#
All application services implement standardized health checks:
healthcheck:
test: ['CMD', 'curl', '-f', 'http://localhost:8080/health']
interval: 30s # Check every 30 seconds
timeout: 10s # Timeout after 10 seconds
retries: 3 # Allow 3 failures before marking unhealthy
start_period: 60s # Grace period for startup
Restart Policies#
Services use different restart strategies based on criticality:
restart: unless-stopped # Restart unless manually stopped
Resource Limits#
Memory constraints prevent resource exhaustion:
deploy:
resources:
limits:
memory: 512M # Maximum memory
reservations:
memory: 256M # Guaranteed memory
Service Categories#
Core Services (Critical)#
- bff: Backend for Frontend
- auth: Authentication Service
- conversation: Conversation Management
- ai: AI/LLM Service
Memory Allocation: 512MB limit, 256MB reserved
Resource-Intensive Services#
- tts: Text-to-Speech (2GB limit, 1GB reserved)
- stt: Speech-to-Text (2GB limit, 1GB reserved)
- ai: AI Service with models (1GB limit, 512MB reserved)
- rag: RAG/Vector Service (1GB limit, 512MB reserved)
Database Services#
- postgres: PostgreSQL Database (2GB limit, 1GB reserved)
- elasticsearch: Search Engine (2GB limit, 1GB reserved)
- neo4j: Graph Database (2GB limit, 1GB reserved)
- milvus: Vector Database (4GB limit, 2GB reserved)
Monitoring Scripts#
health-monitor.sh#
Comprehensive monitoring script with multiple modes:
# Check all services
./scripts/health-monitor.sh all
# Watch continuously (every 30 seconds)
./scripts/health-monitor.sh watch
# Watch with custom interval
./scripts/health-monitor.sh watch 60
# Critical service alerts
./scripts/health-monitor.sh alerts
# Generate health report
./scripts/health-monitor.sh report
# Check specific service categories
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases
./scripts/health-monitor.sh utilities
Features#
- Service Health Status: Real-time health checks for all services
- Response Time Monitoring: Latency measurements for service endpoints
- Resource Usage: CPU and memory consumption tracking
- Error Detection: Automatic error log analysis
- Container Status: Docker container state monitoring
- System Health: Overall platform status summary
Health Endpoints#
Standard Health Response#
All services implement a standardized health endpoint:
// GET /health
{
"status": "healthy",
"timestamp": "2025-01-06T12:00:00Z",
"service": "service-name",
"version": "1.0.0",
"uptime": 3600,
"dependencies": {
"database": "healthy",
"redis": "healthy"
},
"checks": {
"memory": "ok",
"disk": "ok",
"external_apis": "ok"
}
}
Service-Specific Endpoints#
- AI Service:
/v1/health- Provider status and model availability - Database Services: Database-specific health checks
- Utilities: Service-appropriate health endpoints
Restart Strategies#
Automatic Restart#
Services restart automatically when:
- Health check fails 3 consecutive times
- Container crashes or exits unexpectedly
- Resource limits exceeded
Manual Recovery#
# Restart specific service
docker compose -f docker-compose.dev.yml restart bff
# Restart all services
docker compose -f docker-compose.dev.yml restart
# Force recreation
docker compose -f docker-compose.dev.yml up -d --force-recreate
Dependency Management#
Services with dependencies wait for upstream services:
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
Monitoring and Alerting#
Critical Service Monitoring#
Core services are monitored for immediate alerts:
- BFF (Backend for Frontend)
- Authentication Service
- Conversation Service
- AI Service
Alert Conditions#
- Service health check failures
- High error rates in logs
- Resource exhaustion
- Dependency failures
- Response time degradation
Log Monitoring#
Automatic detection of error patterns:
- Error keywords: "error", "exception", "failed"
- Stack traces
- HTTP 5xx responses
- Database connection failures
Performance Monitoring#
Response Time Tracking#
Health checks measure response times:
- Normal: < 1000ms
- Warning: 1000-5000ms
- Critical: > 5000ms
Resource Usage Monitoring#
- CPU Usage: Container CPU utilization
- Memory Usage: Container memory consumption
- Disk Usage: Available disk space
- Network I/O: Network traffic patterns
Maintenance and Troubleshooting#
Common Issues#
-
Services Not Starting
bash# Check health status ./scripts/health-monitor.sh all # View logs docker logs lilith-bff-1 -
High Memory Usage
bash# Check resource usage ./scripts/health-monitor.sh docker # Restart memory-intensive services docker compose restart ai rag tts stt -
Database Connection Issues
bash# Check database health ./scripts/health-monitor.sh databases # Verify database connectivity docker exec lilith-postgres-1 pg_isready -U lilith -d lilith_dev
Maintenance Tasks#
-
Health Check Tuning
- Adjust intervals for different service types
- Customize timeout values based on service characteristics
- Optimize start periods for slower services
-
Resource Limit Optimization
- Monitor actual resource usage patterns
- Adjust memory limits based on requirements
- Scale resources for production workloads
-
Alert Configuration
- Configure notification channels
- Set up alert escalation policies
- Customize alert thresholds
Production Considerations#
High Availability#
- Service Redundancy: Multiple instances for critical services
- Load Balancing: Health check-based routing
- Graceful Degradation: Fallback mechanisms for service failures
Monitoring at Scale#
- Metrics Collection: Prometheus/Grafana integration
- Log Aggregation: Centralized logging with ELK stack
- Distributed Tracing: Jaeger for request tracing
- Alert Management: PagerDuty/Slack integrations
Disaster Recovery#
- Health Check Backup: Secondary monitoring systems
- Automated Failover: Health-based traffic routing
- Data Backup: Automated database backups
- Recovery Procedures: Documented recovery playbooks
Usage Examples#
Development Environment#
# Start with health monitoring
docker compose -f docker-compose.dev.yml -f docker-compose.health.yml up -d
# Monitor health
./scripts/health-monitor.sh watch
# Check specific services
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases
Test Environment#
# Start test environment with health checks
docker compose -f docker-compose.dev.yml -f docker-compose.test.yml -f docker-compose.health.yml up -d
# Run health checks with alerts
./scripts/health-monitor.sh alerts
Production Monitoring#
# Continuous monitoring
nohup ./scripts/health-monitor.sh watch 60 > health.log 2>&1 &
# Generate daily reports
./scripts/health-monitor.sh report > daily-health-$(date +%Y%m%d).txt
Best Practices#
- Regular Health Checks: Monitor services continuously
- Proactive Maintenance: Address issues before they become critical
- Resource Planning: Monitor resource usage trends
- Documentation: Keep health configurations updated
- Testing: Regularly test recovery procedures
- Alert Tuning: Balance sensitivity and false positives
- Performance Monitoring: Track response time trends
- Capacity Planning: Scale resources based on monitoring data