Eve library

Service Recovery Playbook

Runbook

Revision 1

On this page

Purpose: Systematic recovery procedures for application services and microservices architecture failures

Service Recovery Overview#

Service Architecture Context#

text
User Requests → Load Balancer → BFF → Core Services → Data Layer
                                 ↓
                    ┌─────────────┼─────────────┐
                    ↓             ↓             ↓
                  Auth        Conversation    Content
                    ↓             ↓             ↓
                Database      Database      Database

Service Dependencies#

text
Critical Path Services:
├── BFF (Backend-for-Frontend)
│   ├── Depends on: Auth, Conversation, Content, Media
│   └── Required for: All user-facing functionality
├── Auth (Authentication & Authorization)
│   ├── Depends on: Database, Vault
│   └── Required for: All authenticated operations
├── Conversation (Chat & Messaging)
│   ├── Depends on: Auth, Database, AI
│   └── Required for: Core chat functionality
└── Content (Content Management)
    ├── Depends on: Auth, Database, Media
    └── Required for: Content delivery and management

Service Recovery Strategies#

Recovery Priority Matrix#

text
Priority Level | Services | Recovery Order | Max Downtime
P0 (Critical)  | Auth, Database | 1st | 5 minutes
P1 (Core)      | BFF, Conversation | 2nd | 15 minutes
P2 (Important) | Content, Media | 3rd | 30 minutes
P3 (Optional)  | AI, Analytics | 4th | 2 hours

Service Health Indicators#

bash
# Quick health check script
#!/bin/bash
services=("auth" "bff" "conversation" "content" "media" "ai" "notification" "moderation")

for service in "${services[@]}"; do
  echo "Checking $service..."
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://$service.lilith.com/v1/health")
  if [ "$status" -eq 200 ]; then
    echo "✓ $service: Healthy"
  else
    echo "✗ $service: Unhealthy (HTTP $status)"
  fi
done

Individual Service Recovery#

Auth Service Recovery#

Common Auth Failures#

text
1. JWT signing key rotation issues
2. Vault connectivity problems
3. Database connection failures
4. OIDC provider integration issues
5. Session store (Redis) failures

Auth Recovery Process#

bash
# 1. Check auth service pods
kubectl get pods -n lilith-auth -o wide
kubectl logs deployment/auth-deployment -n lilith-auth --tail=50

# 2. Check dependencies
# Vault connectivity
kubectl exec -it auth-pod -n lilith-auth -- curl -f $VAULT_URL/v1/sys/health

# Database connectivity
kubectl exec -it auth-pod -n lilith-auth -- nc -z postgres-service 5432

# 3. Check configuration
kubectl get configmap auth-config -n lilith-auth -o yaml
kubectl get secret auth-secrets -n lilith-auth -o yaml

# 4. Restart with fresh configuration
kubectl rollout restart deployment auth-deployment -n lilith-auth
kubectl wait --for=condition=Available deployment auth-deployment -n lilith-auth --timeout=300s

# 5. Validate functionality
curl -X POST https://auth.lilith.com/v1/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@lilith.com","password":"testpass"}'

Auth-Specific Recovery Actions#

bash
# Rotate JWT signing keys if compromised
kubectl exec -it auth-pod -n lilith-auth -- \
  /app/scripts/rotate-jwt-keys.sh

# Clear session cache if corrupted
kubectl exec -it redis-session-store -- redis-cli FLUSHDB

# Restart OIDC provider connections
kubectl patch deployment auth-deployment -n lilith-auth \
  --patch='{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'$(date +%Y-%m-%dT%H:%M:%S%z)'"}}}}}'

BFF Service Recovery#

BFF Failure Symptoms#

text
- Gateway timeouts (504 errors)
- Service discovery failures
- Circuit breaker activation
- Rate limiting issues
- Load balancer health check failures

BFF Recovery Process#

bash
# 1. Check BFF service status
kubectl get pods -n lilith-bff
kubectl describe pods -n lilith-bff

# 2. Check downstream service connectivity
kubectl exec -it bff-pod -n lilith-bff -- \
  for service in auth conversation content; do
    echo "Testing $service..."
    curl -f "http://$service.lilith-$service.svc.cluster.local/v1/health"
  done

# 3. Check load balancer configuration
kubectl get ingress -n lilith-bff
kubectl describe ingress main-ingress -n lilith-bff

# 4. Scale up if capacity issue
kubectl scale deployment bff-deployment --replicas=10 -n lilith-bff

# 5. Rolling restart to clear any stuck connections
kubectl rollout restart deployment bff-deployment -n lilith-bff
kubectl rollout status deployment bff-deployment -n lilith-bff

BFF Circuit Breaker Reset#

bash
# Reset circuit breakers if they're preventing recovery
kubectl exec -it bff-pod -n lilith-bff -- \
  curl -X POST "http://localhost:8080/admin/circuit-breakers/reset"

# Check circuit breaker status
kubectl exec -it bff-pod -n lilith-bff -- \
  curl "http://localhost:8080/admin/circuit-breakers/status"

Conversation Service Recovery#

Conversation Failure Scenarios#

text
- Message persistence failures
- WebSocket connection issues
- Event streaming problems
- Chat history corruption
- Real-time notification failures

Conversation Recovery Steps#

bash
# 1. Check conversation service health
kubectl get pods -n lilith-conversation
kubectl logs deployment/conversation-deployment -n lilith-conversation --tail=100

# 2. Check database connectivity specific to conversations
kubectl exec -it conversation-pod -n lilith-conversation -- \
  psql -h postgres-service -d lilith -c "SELECT count(*) FROM conversations LIMIT 1;"

# 3. Check event streaming (if using Kafka/Redis)
kubectl exec -it conversation-pod -n lilith-conversation -- \
  /app/scripts/check-event-stream.sh

# 4. Restart service with zero-downtime deployment
kubectl patch deployment conversation-deployment -n lilith-conversation \
  --patch='{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}'

kubectl rollout restart deployment conversation-deployment -n lilith-conversation

# 5. Validate chat functionality
curl -X POST https://conversation.lilith.com/v1/conversations \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message":"Test message"}'

Multi-Service Failure Recovery#

Cascading Failure Recovery#

Scenario: Database Failure Causing Service Cascade#

text
Database Down → Auth Fails → BFF Can't Authenticate → All Services Inaccessible

Recovery Sequence#

bash
# 1. Restore database first (highest priority)
./scripts/database-recovery.sh

# 2. Wait for database to be fully operational
kubectl wait --for=condition=Ready pod postgres-primary-0 -n database --timeout=600s

# 3. Restart services in dependency order
services=("auth" "bff" "conversation" "content" "media")
for service in "${services[@]}"; do
  echo "Restarting $service..."
  kubectl rollout restart deployment ${service}-deployment -n lilith-${service}
  kubectl wait --for=condition=Available deployment ${service}-deployment -n lilith-${service} --timeout=300s

  # Validate service before proceeding
  health_check_url="https://${service}.lilith.com/v1/health"
  while ! curl -f "$health_check_url" > /dev/null 2>&1; do
    echo "Waiting for $service to be healthy..."
    sleep 10
  done
  echo "✓ $service is healthy"
done

Resource Exhaustion Recovery#

Scenario: Cluster Resource Exhaustion#

bash
# 1. Identify resource constraints
kubectl top nodes
kubectl describe nodes | grep -A10 "Allocated resources"

# 2. Scale down non-critical services temporarily
kubectl scale deployment ai-deployment --replicas=0 -n lilith-ai
kubectl scale deployment analytics-deployment --replicas=0 -n lilith-analytics

# 3. Add cluster capacity if possible
./scripts/scale-cluster-nodes.sh --increase 2

# 4. Restart critical services with resource limits
kubectl apply -f manifests/critical-services-reduced-resources.yaml

# 5. Monitor and gradually restore non-critical services
for service in ai analytics; do
  kubectl scale deployment ${service}-deployment --replicas=1 -n lilith-${service}
  sleep 60  # Allow time for resources to stabilize
done

Service-Specific Recovery Procedures#

AI Service Recovery#

AI Service Dependencies#

text
- Model artifacts storage (S3/GCS)
- GPU resources
- External API providers (OpenAI, Anthropic)
- Vector database connectivity
- Model serving infrastructure

AI Recovery Process#

bash
# 1. Check GPU resource availability
kubectl get nodes -l node-type=gpu -o wide
kubectl describe nodes | grep -A5 "nvidia.com/gpu"

# 2. Check model storage accessibility
kubectl exec -it ai-pod -n lilith-ai -- \
  aws s3 ls s3://lilith-models/current/

# 3. Validate external API connectivity
kubectl exec -it ai-pod -n lilith-ai -- \
  curl -H "Authorization: Bearer $OPENAI_API_KEY" \
  "https://api.openai.com/v1/models"

# 4. Restart AI services with model reload
kubectl delete pods -l app=ai -n lilith-ai
kubectl wait --for=condition=Ready pod -l app=ai -n lilith-ai --timeout=600s

# 5. Run model inference test
curl -X POST https://ai.lilith.com/v1/generate \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -d '{"prompt":"Test inference","max_tokens":10}'

Media Service Recovery#

Media Service Components#

text
- File upload/download APIs
- Image/video processing pipelines
- CDN integration
- Storage backend (S3/GCS)
- Transcoding services

Media Recovery Steps#

bash
# 1. Check storage backend connectivity
kubectl exec -it media-pod -n lilith-media -- \
  aws s3 ls s3://lilith-media-storage/

# 2. Check CDN health
kubectl exec -it media-pod -n lilith-media -- \
  curl -I "https://cdn.lilith.com/health"

# 3. Restart media processing workers
kubectl rollout restart deployment media-worker -n lilith-media
kubectl rollout restart deployment media-transcoder -n lilith-media

# 4. Test file upload functionality
curl -X POST https://media.lilith.com/v1/upload \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -F "file=@test-image.jpg"

Notification Service Recovery#

Notification Dependencies#

text
- Push notification providers (APNS, FCM)
- Email service providers (SendGrid, SES)
- Message queues (Redis, Kafka)
- Template storage
- User preference database

Notification Recovery Process#

bash
# 1. Check message queue health
kubectl exec -it notification-pod -n lilith-notification -- \
  redis-cli -h redis-service ping

# 2. Test notification providers
kubectl exec -it notification-pod -n lilith-notification -- \
  /app/scripts/test-notification-providers.sh

# 3. Clear stuck message queues if needed
kubectl exec -it notification-pod -n lilith-notification -- \
  redis-cli -h redis-service FLUSHDB

# 4. Restart notification workers
kubectl rollout restart deployment notification-worker -n lilith-notification

# 5. Test notification delivery
curl -X POST https://notification.lilith.com/v1/send \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -d '{"type":"test","user_id":"test-user","message":"Test notification"}'

Service Recovery Automation#

Automated Recovery Scripts#

Self-Healing Service Monitor#

bash
#!/bin/bash
# Service health monitor with automatic recovery

SERVICES=("auth" "bff" "conversation" "content" "media")
RECOVERY_LOG="/var/log/service-recovery.log"

monitor_and_recover() {
  local service=$1
  local namespace="lilith-${service}"

  # Check service health
  health_url="https://${service}.lilith.com/v1/health"

  if ! curl -f "$health_url" > /dev/null 2>&1; then
    echo "$(date): $service unhealthy, attempting recovery" >> $RECOVERY_LOG

    # Attempt recovery
    kubectl rollout restart deployment ${service}-deployment -n $namespace

    # Wait for recovery
    sleep 60

    # Verify recovery
    if curl -f "$health_url" > /dev/null 2>&1; then
      echo "$(date): $service recovery successful" >> $RECOVERY_LOG
      # Send success notification
      /scripts/send-alert.sh "✓ $service automatically recovered"
    else
      echo "$(date): $service recovery failed, escalating" >> $RECOVERY_LOG
      # Escalate to on-call
      /scripts/escalate-incident.sh "$service automatic recovery failed"
    fi
  fi
}

# Monitor all services
for service in "${SERVICES[@]}"; do
  monitor_and_recover "$service" &
done

wait

Dependency-Aware Recovery#

bash
#!/bin/bash
# Recovery script that respects service dependencies

declare -A SERVICE_DEPENDENCIES
SERVICE_DEPENDENCIES[auth]="database vault"
SERVICE_DEPENDENCIES[bff]="auth"
SERVICE_DEPENDENCIES[conversation]="auth database"
SERVICE_DEPENDENCIES[content]="auth database"
SERVICE_DEPENDENCIES[media]="auth storage"

recover_service_with_dependencies() {
  local service=$1
  local dependencies="${SERVICE_DEPENDENCIES[$service]}"

  echo "Recovering $service with dependencies: $dependencies"

  # First, ensure all dependencies are healthy
  for dep in $dependencies; do
    if ! check_service_health "$dep"; then
      echo "Dependency $dep is unhealthy, recovering first..."
      recover_service_with_dependencies "$dep"
    fi
  done

  # Now recover the service itself
  echo "All dependencies healthy, recovering $service..."
  kubectl rollout restart deployment ${service}-deployment -n lilith-${service}
  kubectl wait --for=condition=Available deployment ${service}-deployment -n lilith-${service} --timeout=300s

  # Validate recovery
  if check_service_health "$service"; then
    echo "✓ $service recovered successfully"
  else
    echo "✗ $service recovery failed"
    return 1
  fi
}

check_service_health() {
  local service=$1
  curl -f "https://${service}.lilith.com/v1/health" > /dev/null 2>&1
}

Service Recovery Testing#

Chaos Engineering for Recovery Testing#

bash
#!/bin/bash
# Chaos engineering script to test service recovery

chaos_test_service_failure() {
  local service=$1
  local test_duration=${2:-300}  # 5 minutes default

  echo "Starting chaos test: $service failure for ${test_duration}s"

  # Record baseline metrics
  kubectl top pods -n lilith-${service} > /tmp/baseline-${service}.txt

  # Introduce failure (scale to 0)
  kubectl scale deployment ${service}-deployment --replicas=0 -n lilith-${service}

  echo "Service $service scaled down, monitoring recovery..."

  # Wait for automatic recovery or manual intervention
  sleep $test_duration

  # Check if service recovered
  if [ $(kubectl get deployment ${service}-deployment -n lilith-${service} -o jsonpath='{.spec.replicas}') -gt 0 ]; then
    echo "✓ $service recovered within test duration"
  else
    echo "✗ $service did not recover, manual intervention required"
    # Restore service
    kubectl scale deployment ${service}-deployment --replicas=2 -n lilith-${service}
  fi

  # Record post-test metrics
  kubectl top pods -n lilith-${service} > /tmp/posttest-${service}.txt

  echo "Chaos test completed for $service"
}

# Run chaos tests for all services
for service in auth bff conversation content; do
  chaos_test_service_failure "$service" 180
  sleep 300  # Wait between tests
done

Recovery Performance Metrics#

bash
#!/bin/bash
# Measure service recovery performance

measure_recovery_time() {
  local service=$1

  echo "Measuring recovery time for $service..."

  start_time=$(date +%s)

  # Simulate failure
  kubectl delete pods -l app=$service -n lilith-$service

  # Wait for recovery
  kubectl wait --for=condition=Ready pod -l app=$service -n lilith-$service --timeout=600s

  # Wait for service to be healthy
  while ! curl -f "https://${service}.lilith.com/v1/health" > /dev/null 2>&1; do
    sleep 5
  done

  end_time=$(date +%s)
  recovery_time=$((end_time - start_time))

  echo "$service recovery time: ${recovery_time}s"
  echo "$(date),$service,$recovery_time" >> /var/log/recovery-metrics.csv
}

# Test all critical services
for service in auth bff conversation; do
  measure_recovery_time "$service"
  sleep 60  # Wait between tests
done

Remember: Service recovery should follow dependency order, validate each step, and maintain comprehensive logging for post-incident analysis. Always test recovery procedures in non-production environments first.