Purpose: Systematic recovery procedures for application services and microservices architecture failures
Service Recovery Overview#
Service Architecture Context#
text
User Requests → Load Balancer → BFF → Core Services → Data Layer
↓
┌─────────────┼─────────────┐
↓ ↓ ↓
Auth Conversation Content
↓ ↓ ↓
Database Database Database
Service Dependencies#
text
Critical Path Services:
├── BFF (Backend-for-Frontend)
│ ├── Depends on: Auth, Conversation, Content, Media
│ └── Required for: All user-facing functionality
├── Auth (Authentication & Authorization)
│ ├── Depends on: Database, Vault
│ └── Required for: All authenticated operations
├── Conversation (Chat & Messaging)
│ ├── Depends on: Auth, Database, AI
│ └── Required for: Core chat functionality
└── Content (Content Management)
├── Depends on: Auth, Database, Media
└── Required for: Content delivery and management
Service Recovery Strategies#
Recovery Priority Matrix#
text
Priority Level | Services | Recovery Order | Max Downtime
P0 (Critical) | Auth, Database | 1st | 5 minutes
P1 (Core) | BFF, Conversation | 2nd | 15 minutes
P2 (Important) | Content, Media | 3rd | 30 minutes
P3 (Optional) | AI, Analytics | 4th | 2 hours
Service Health Indicators#
bash
# Quick health check script
#!/bin/bash
services=("auth" "bff" "conversation" "content" "media" "ai" "notification" "moderation")
for service in "${services[@]}"; do
echo "Checking $service..."
status=$(curl -s -o /dev/null -w "%{http_code}" "https://$service.lilith.com/v1/health")
if [ "$status" -eq 200 ]; then
echo "✓ $service: Healthy"
else
echo "✗ $service: Unhealthy (HTTP $status)"
fi
done
Individual Service Recovery#
Auth Service Recovery#
Common Auth Failures#
text
1. JWT signing key rotation issues
2. Vault connectivity problems
3. Database connection failures
4. OIDC provider integration issues
5. Session store (Redis) failures
Auth Recovery Process#
bash
# 1. Check auth service pods
kubectl get pods -n lilith-auth -o wide
kubectl logs deployment/auth-deployment -n lilith-auth --tail=50
# 2. Check dependencies
# Vault connectivity
kubectl exec -it auth-pod -n lilith-auth -- curl -f $VAULT_URL/v1/sys/health
# Database connectivity
kubectl exec -it auth-pod -n lilith-auth -- nc -z postgres-service 5432
# 3. Check configuration
kubectl get configmap auth-config -n lilith-auth -o yaml
kubectl get secret auth-secrets -n lilith-auth -o yaml
# 4. Restart with fresh configuration
kubectl rollout restart deployment auth-deployment -n lilith-auth
kubectl wait --for=condition=Available deployment auth-deployment -n lilith-auth --timeout=300s
# 5. Validate functionality
curl -X POST https://auth.lilith.com/v1/login \
-H "Content-Type: application/json" \
-d '{"email":"test@lilith.com","password":"testpass"}'
Auth-Specific Recovery Actions#
bash
# Rotate JWT signing keys if compromised
kubectl exec -it auth-pod -n lilith-auth -- \
/app/scripts/rotate-jwt-keys.sh
# Clear session cache if corrupted
kubectl exec -it redis-session-store -- redis-cli FLUSHDB
# Restart OIDC provider connections
kubectl patch deployment auth-deployment -n lilith-auth \
--patch='{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'$(date +%Y-%m-%dT%H:%M:%S%z)'"}}}}}'
BFF Service Recovery#
BFF Failure Symptoms#
text
- Gateway timeouts (504 errors)
- Service discovery failures
- Circuit breaker activation
- Rate limiting issues
- Load balancer health check failures
BFF Recovery Process#
bash
# 1. Check BFF service status
kubectl get pods -n lilith-bff
kubectl describe pods -n lilith-bff
# 2. Check downstream service connectivity
kubectl exec -it bff-pod -n lilith-bff -- \
for service in auth conversation content; do
echo "Testing $service..."
curl -f "http://$service.lilith-$service.svc.cluster.local/v1/health"
done
# 3. Check load balancer configuration
kubectl get ingress -n lilith-bff
kubectl describe ingress main-ingress -n lilith-bff
# 4. Scale up if capacity issue
kubectl scale deployment bff-deployment --replicas=10 -n lilith-bff
# 5. Rolling restart to clear any stuck connections
kubectl rollout restart deployment bff-deployment -n lilith-bff
kubectl rollout status deployment bff-deployment -n lilith-bff
BFF Circuit Breaker Reset#
bash
# Reset circuit breakers if they're preventing recovery
kubectl exec -it bff-pod -n lilith-bff -- \
curl -X POST "http://localhost:8080/admin/circuit-breakers/reset"
# Check circuit breaker status
kubectl exec -it bff-pod -n lilith-bff -- \
curl "http://localhost:8080/admin/circuit-breakers/status"
Conversation Service Recovery#
Conversation Failure Scenarios#
text
- Message persistence failures
- WebSocket connection issues
- Event streaming problems
- Chat history corruption
- Real-time notification failures
Conversation Recovery Steps#
bash
# 1. Check conversation service health
kubectl get pods -n lilith-conversation
kubectl logs deployment/conversation-deployment -n lilith-conversation --tail=100
# 2. Check database connectivity specific to conversations
kubectl exec -it conversation-pod -n lilith-conversation -- \
psql -h postgres-service -d lilith -c "SELECT count(*) FROM conversations LIMIT 1;"
# 3. Check event streaming (if using Kafka/Redis)
kubectl exec -it conversation-pod -n lilith-conversation -- \
/app/scripts/check-event-stream.sh
# 4. Restart service with zero-downtime deployment
kubectl patch deployment conversation-deployment -n lilith-conversation \
--patch='{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}'
kubectl rollout restart deployment conversation-deployment -n lilith-conversation
# 5. Validate chat functionality
curl -X POST https://conversation.lilith.com/v1/conversations \
-H "Authorization: Bearer $TEST_TOKEN" \
-H "Content-Type: application/json" \
-d '{"message":"Test message"}'
Multi-Service Failure Recovery#
Cascading Failure Recovery#
Scenario: Database Failure Causing Service Cascade#
text
Database Down → Auth Fails → BFF Can't Authenticate → All Services Inaccessible
Recovery Sequence#
bash
# 1. Restore database first (highest priority)
./scripts/database-recovery.sh
# 2. Wait for database to be fully operational
kubectl wait --for=condition=Ready pod postgres-primary-0 -n database --timeout=600s
# 3. Restart services in dependency order
services=("auth" "bff" "conversation" "content" "media")
for service in "${services[@]}"; do
echo "Restarting $service..."
kubectl rollout restart deployment ${service}-deployment -n lilith-${service}
kubectl wait --for=condition=Available deployment ${service}-deployment -n lilith-${service} --timeout=300s
# Validate service before proceeding
health_check_url="https://${service}.lilith.com/v1/health"
while ! curl -f "$health_check_url" > /dev/null 2>&1; do
echo "Waiting for $service to be healthy..."
sleep 10
done
echo "✓ $service is healthy"
done
Resource Exhaustion Recovery#
Scenario: Cluster Resource Exhaustion#
bash
# 1. Identify resource constraints
kubectl top nodes
kubectl describe nodes | grep -A10 "Allocated resources"
# 2. Scale down non-critical services temporarily
kubectl scale deployment ai-deployment --replicas=0 -n lilith-ai
kubectl scale deployment analytics-deployment --replicas=0 -n lilith-analytics
# 3. Add cluster capacity if possible
./scripts/scale-cluster-nodes.sh --increase 2
# 4. Restart critical services with resource limits
kubectl apply -f manifests/critical-services-reduced-resources.yaml
# 5. Monitor and gradually restore non-critical services
for service in ai analytics; do
kubectl scale deployment ${service}-deployment --replicas=1 -n lilith-${service}
sleep 60 # Allow time for resources to stabilize
done
Service-Specific Recovery Procedures#
AI Service Recovery#
AI Service Dependencies#
text
- Model artifacts storage (S3/GCS)
- GPU resources
- External API providers (OpenAI, Anthropic)
- Vector database connectivity
- Model serving infrastructure
AI Recovery Process#
bash
# 1. Check GPU resource availability
kubectl get nodes -l node-type=gpu -o wide
kubectl describe nodes | grep -A5 "nvidia.com/gpu"
# 2. Check model storage accessibility
kubectl exec -it ai-pod -n lilith-ai -- \
aws s3 ls s3://lilith-models/current/
# 3. Validate external API connectivity
kubectl exec -it ai-pod -n lilith-ai -- \
curl -H "Authorization: Bearer $OPENAI_API_KEY" \
"https://api.openai.com/v1/models"
# 4. Restart AI services with model reload
kubectl delete pods -l app=ai -n lilith-ai
kubectl wait --for=condition=Ready pod -l app=ai -n lilith-ai --timeout=600s
# 5. Run model inference test
curl -X POST https://ai.lilith.com/v1/generate \
-H "Authorization: Bearer $TEST_TOKEN" \
-d '{"prompt":"Test inference","max_tokens":10}'
Media Service Recovery#
Media Service Components#
text
- File upload/download APIs
- Image/video processing pipelines
- CDN integration
- Storage backend (S3/GCS)
- Transcoding services
Media Recovery Steps#
bash
# 1. Check storage backend connectivity
kubectl exec -it media-pod -n lilith-media -- \
aws s3 ls s3://lilith-media-storage/
# 2. Check CDN health
kubectl exec -it media-pod -n lilith-media -- \
curl -I "https://cdn.lilith.com/health"
# 3. Restart media processing workers
kubectl rollout restart deployment media-worker -n lilith-media
kubectl rollout restart deployment media-transcoder -n lilith-media
# 4. Test file upload functionality
curl -X POST https://media.lilith.com/v1/upload \
-H "Authorization: Bearer $TEST_TOKEN" \
-F "file=@test-image.jpg"
Notification Service Recovery#
Notification Dependencies#
text
- Push notification providers (APNS, FCM)
- Email service providers (SendGrid, SES)
- Message queues (Redis, Kafka)
- Template storage
- User preference database
Notification Recovery Process#
bash
# 1. Check message queue health
kubectl exec -it notification-pod -n lilith-notification -- \
redis-cli -h redis-service ping
# 2. Test notification providers
kubectl exec -it notification-pod -n lilith-notification -- \
/app/scripts/test-notification-providers.sh
# 3. Clear stuck message queues if needed
kubectl exec -it notification-pod -n lilith-notification -- \
redis-cli -h redis-service FLUSHDB
# 4. Restart notification workers
kubectl rollout restart deployment notification-worker -n lilith-notification
# 5. Test notification delivery
curl -X POST https://notification.lilith.com/v1/send \
-H "Authorization: Bearer $TEST_TOKEN" \
-d '{"type":"test","user_id":"test-user","message":"Test notification"}'
Service Recovery Automation#
Automated Recovery Scripts#
Self-Healing Service Monitor#
bash
#!/bin/bash
# Service health monitor with automatic recovery
SERVICES=("auth" "bff" "conversation" "content" "media")
RECOVERY_LOG="/var/log/service-recovery.log"
monitor_and_recover() {
local service=$1
local namespace="lilith-${service}"
# Check service health
health_url="https://${service}.lilith.com/v1/health"
if ! curl -f "$health_url" > /dev/null 2>&1; then
echo "$(date): $service unhealthy, attempting recovery" >> $RECOVERY_LOG
# Attempt recovery
kubectl rollout restart deployment ${service}-deployment -n $namespace
# Wait for recovery
sleep 60
# Verify recovery
if curl -f "$health_url" > /dev/null 2>&1; then
echo "$(date): $service recovery successful" >> $RECOVERY_LOG
# Send success notification
/scripts/send-alert.sh "✓ $service automatically recovered"
else
echo "$(date): $service recovery failed, escalating" >> $RECOVERY_LOG
# Escalate to on-call
/scripts/escalate-incident.sh "$service automatic recovery failed"
fi
fi
}
# Monitor all services
for service in "${SERVICES[@]}"; do
monitor_and_recover "$service" &
done
wait
Dependency-Aware Recovery#
bash
#!/bin/bash
# Recovery script that respects service dependencies
declare -A SERVICE_DEPENDENCIES
SERVICE_DEPENDENCIES[auth]="database vault"
SERVICE_DEPENDENCIES[bff]="auth"
SERVICE_DEPENDENCIES[conversation]="auth database"
SERVICE_DEPENDENCIES[content]="auth database"
SERVICE_DEPENDENCIES[media]="auth storage"
recover_service_with_dependencies() {
local service=$1
local dependencies="${SERVICE_DEPENDENCIES[$service]}"
echo "Recovering $service with dependencies: $dependencies"
# First, ensure all dependencies are healthy
for dep in $dependencies; do
if ! check_service_health "$dep"; then
echo "Dependency $dep is unhealthy, recovering first..."
recover_service_with_dependencies "$dep"
fi
done
# Now recover the service itself
echo "All dependencies healthy, recovering $service..."
kubectl rollout restart deployment ${service}-deployment -n lilith-${service}
kubectl wait --for=condition=Available deployment ${service}-deployment -n lilith-${service} --timeout=300s
# Validate recovery
if check_service_health "$service"; then
echo "✓ $service recovered successfully"
else
echo "✗ $service recovery failed"
return 1
fi
}
check_service_health() {
local service=$1
curl -f "https://${service}.lilith.com/v1/health" > /dev/null 2>&1
}
Service Recovery Testing#
Chaos Engineering for Recovery Testing#
bash
#!/bin/bash
# Chaos engineering script to test service recovery
chaos_test_service_failure() {
local service=$1
local test_duration=${2:-300} # 5 minutes default
echo "Starting chaos test: $service failure for ${test_duration}s"
# Record baseline metrics
kubectl top pods -n lilith-${service} > /tmp/baseline-${service}.txt
# Introduce failure (scale to 0)
kubectl scale deployment ${service}-deployment --replicas=0 -n lilith-${service}
echo "Service $service scaled down, monitoring recovery..."
# Wait for automatic recovery or manual intervention
sleep $test_duration
# Check if service recovered
if [ $(kubectl get deployment ${service}-deployment -n lilith-${service} -o jsonpath='{.spec.replicas}') -gt 0 ]; then
echo "✓ $service recovered within test duration"
else
echo "✗ $service did not recover, manual intervention required"
# Restore service
kubectl scale deployment ${service}-deployment --replicas=2 -n lilith-${service}
fi
# Record post-test metrics
kubectl top pods -n lilith-${service} > /tmp/posttest-${service}.txt
echo "Chaos test completed for $service"
}
# Run chaos tests for all services
for service in auth bff conversation content; do
chaos_test_service_failure "$service" 180
sleep 300 # Wait between tests
done
Recovery Performance Metrics#
bash
#!/bin/bash
# Measure service recovery performance
measure_recovery_time() {
local service=$1
echo "Measuring recovery time for $service..."
start_time=$(date +%s)
# Simulate failure
kubectl delete pods -l app=$service -n lilith-$service
# Wait for recovery
kubectl wait --for=condition=Ready pod -l app=$service -n lilith-$service --timeout=600s
# Wait for service to be healthy
while ! curl -f "https://${service}.lilith.com/v1/health" > /dev/null 2>&1; do
sleep 5
done
end_time=$(date +%s)
recovery_time=$((end_time - start_time))
echo "$service recovery time: ${recovery_time}s"
echo "$(date),$service,$recovery_time" >> /var/log/recovery-metrics.csv
}
# Test all critical services
for service in auth bff conversation; do
measure_recovery_time "$service"
sleep 60 # Wait between tests
done
Remember: Service recovery should follow dependency order, validate each step, and maintain comprehensive logging for post-incident analysis. Always test recovery procedures in non-production environments first.