Eve library

Deployment Procedures

Runbook

Revision 1

On this page

Purpose: Safe deployment practices for Lilith microservices using GitOps and rolling updates

Deployment Architecture Overview#

GitOps Workflow#

text
Developer → Git Push → CI Pipeline → Image Build → GitOps Repo Update → ArgoCD → Kubernetes Deployment
    ↓           ↓           ↓             ↓              ↓              ↓              ↓
  Feature    Trigger     Tests        Container       Manifest      Sync         Rolling
   Branch    Build      Pass         Registry        Update       Deploy        Update

Deployment Environments#

text
Development → Staging → Production
     ↓           ↓           ↓
  Feature    Integration  Production
  Testing    Testing      Workloads
     ↓           ↓           ↓
  Immediate  Automated    Manual
  Deploy     Deploy       Approval

Pre-Deployment Checklist#

Code Review Requirements#

  • Code review approved by at least 2 engineers
  • All tests passing (unit, integration, e2e)
  • Security scan completed without high/critical issues
  • Documentation updated for API changes
  • Database migrations tested in staging

Infrastructure Readiness#

  • Target environment healthy (all services green)
  • Resource capacity sufficient for new deployment
  • Database migrations completed if required
  • External dependencies verified
  • Monitoring and alerting configured

Deployment Planning#

  • Deployment window scheduled (avoid peak hours)
  • Rollback plan documented and tested
  • Feature flags configured for gradual rollout
  • Communication plan for user-facing changes
  • On-call engineer available during deployment

Deployment Types and Procedures#

Standard Rolling Deployment#

For Stateless Services (BFF, Auth, Conversation)#

bash
# 1. Verify current deployment health
kubectl get deployments -n lilith-bff
kubectl rollout status deployment bff-deployment -n lilith-bff

# 2. Update image tag in ArgoCD application
argocd app set lilith-bff --parameter image.tag=v2.1.3

# 3. Sync deployment
argocd app sync lilith-bff

# 4. Monitor rollout progress
kubectl rollout status deployment bff-deployment -n lilith-bff --timeout=600s

# 5. Verify health after deployment
curl -f https://bff.lilith.com/v1/health
./scripts/post-deploy-validation.sh bff

Rolling Update Configuration#

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bff-deployment
  namespace: lilith-bff
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1 # Only 1 pod down at a time
      maxSurge: 2 # Up to 2 extra pods during update
  template:
    spec:
      containers:
        - name: bff
          image: lilith/bff:v2.1.3
          readinessProbe:
            httpGet:
              path: /v1/health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /v1/health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 30

Blue-Green Deployment#

For Critical Services Requiring Zero Downtime#

bash
# 1. Deploy to green environment (inactive)
kubectl apply -f manifests/bff-green-deployment.yaml

# 2. Wait for green deployment to be ready
kubectl wait --for=condition=Available deployment bff-green-deployment -n lilith-bff --timeout=600s

# 3. Run validation tests against green environment
./scripts/validate-green-deployment.sh bff

# 4. Switch traffic to green (update service selector)
kubectl patch service bff-service -n lilith-bff \
  --patch='{"spec":{"selector":{"version":"green"}}}'

# 5. Monitor for any issues
sleep 300
./scripts/monitor-deployment-health.sh bff 300

# 6. If successful, scale down blue deployment
kubectl scale deployment bff-blue-deployment --replicas=0 -n lilith-bff

# 7. If issues detected, rollback immediately
# kubectl patch service bff-service -n lilith-bff \
#   --patch='{"spec":{"selector":{"version":"blue"}}}'

Canary Deployment#

For Gradual Feature Rollouts#

bash
# 1. Deploy canary version (5% traffic)
kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bff-rollout
  namespace: lilith-bff
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - setWeight: 20
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100
      canaryService: bff-canary-service
      stableService: bff-stable-service
      trafficRouting:
        istio:
          virtualService:
            name: bff-vs
            routes:
            - primary
  selector:
    matchLabels:
      app: bff
  template:
    spec:
      containers:
      - name: bff
        image: lilith/bff:v2.1.3-canary
EOF

# 2. Monitor canary metrics
kubectl argo rollouts get rollout bff-rollout -n lilith-bff --watch

# 3. Promote if metrics look good, or abort if issues
# kubectl argo rollouts promote bff-rollout -n lilith-bff
# kubectl argo rollouts abort bff-rollout -n lilith-bff

Database Migrations#

Safe Migration Procedures#

Pre-Migration Validation#

bash
# 1. Backup database before migration
kubectl exec -it postgres-primary-0 -n database -- \
  pg_dump lilith > /backups/pre-migration-$(date +%Y%m%d-%H%M).sql

# 2. Test migration on staging database
kubectl exec -it postgres-staging-0 -n staging -- \
  psql lilith < migrations/v2.1.3-migration.sql

# 3. Validate migration results
kubectl exec -it postgres-staging-0 -n staging -- \
  psql lilith -c "SELECT count(*) FROM new_table;"

# 4. Test application compatibility with new schema
./scripts/test-app-with-new-schema.sh staging

Production Migration Execution#

bash
# 1. Put application in maintenance mode (if required)
kubectl scale deployment --replicas=0 -n lilith-bff
kubectl apply -f manifests/maintenance-page.yaml

# 2. Run migration with monitoring
kubectl exec -it postgres-primary-0 -n database -- \
  psql lilith < migrations/v2.1.3-migration.sql | tee migration.log

# 3. Validate migration success
kubectl exec -it postgres-primary-0 -n database -- \
  psql lilith -c "\d+ new_table;"

# 4. Deploy application with new schema support
kubectl scale deployment bff-deployment --replicas=6 -n lilith-bff
kubectl wait --for=condition=Available deployment bff-deployment -n lilith-bff --timeout=600s

# 5. Remove maintenance mode
kubectl delete -f manifests/maintenance-page.yaml

Backward-Compatible Migration Pattern#

sql
-- Phase 1: Add new column (nullable)
ALTER TABLE users ADD COLUMN new_feature_flag BOOLEAN DEFAULT NULL;

-- Deploy application version that can handle both old and new schema

-- Phase 2: Backfill data (in next deployment)
UPDATE users SET new_feature_flag = false WHERE new_feature_flag IS NULL;

-- Phase 3: Make column NOT NULL (in subsequent deployment)
ALTER TABLE users ALTER COLUMN new_feature_flag SET NOT NULL;
ALTER TABLE users ALTER COLUMN new_feature_flag SET DEFAULT false;

Rollback Procedures#

Automated Rollback Triggers#

bash
#!/bin/bash
# Automated rollback script based on health metrics

SERVICE=$1
NAMESPACE="lilith-${SERVICE}"
ROLLBACK_THRESHOLD_ERROR_RATE=5.0
ROLLBACK_THRESHOLD_RESPONSE_TIME=5000

# Get current error rate and response time
error_rate=$(kubectl exec -n monitoring prometheus-0 -- \
  promtool query instant "rate(http_requests_total{job=\"$SERVICE\",status=~\"5..\"}[5m]) * 100")

response_time=$(kubectl exec -n monitoring prometheus-0 -- \
  promtool query instant "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"$SERVICE\"}[5m])) * 1000")

# Check if rollback is needed
if (( $(echo "$error_rate > $ROLLBACK_THRESHOLD_ERROR_RATE" | bc -l) )) || \
   (( $(echo "$response_time > $ROLLBACK_THRESHOLD_RESPONSE_TIME" | bc -l) )); then

  echo "Rollback triggered for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"

  # Execute rollback
  kubectl rollout undo deployment ${SERVICE}-deployment -n $NAMESPACE

  # Wait for rollback to complete
  kubectl rollout status deployment ${SERVICE}-deployment -n $NAMESPACE --timeout=300s

  # Notify team
  ./scripts/send-alert.sh "🔄 Automatic rollback executed for $SERVICE due to performance degradation"

  # Create incident for investigation
  ./scripts/create-incident.sh "Automatic rollback: $SERVICE performance degradation"
else
  echo "No rollback needed for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"
fi

Manual Rollback Process#

bash
# 1. Identify current and previous versions
kubectl rollout history deployment bff-deployment -n lilith-bff

# 2. Rollback to previous version
kubectl rollout undo deployment bff-deployment -n lilith-bff

# 3. Or rollback to specific revision
kubectl rollout undo deployment bff-deployment --to-revision=3 -n lilith-bff

# 4. Monitor rollback progress
kubectl rollout status deployment bff-deployment -n lilith-bff

# 5. Verify service health after rollback
curl -f https://bff.lilith.com/v1/health
./scripts/post-rollback-validation.sh bff

# 6. Update ArgoCD to match rolled-back state
argocd app set lilith-bff --parameter image.tag=v2.1.2

Feature Flag Management#

Feature Flag Deployment Pattern#

bash
# 1. Deploy new feature with flag disabled
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: lilith-bff
data:
  NEW_CHAT_INTERFACE: "false"
  ADVANCED_SEARCH: "false"
  VOICE_CHAT_V2: "false"
EOF

# 2. Deploy application code that checks feature flags
kubectl set image deployment/bff-deployment bff=lilith/bff:v2.1.3-with-flags -n lilith-bff

# 3. Gradually enable feature for user segments
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"true:beta-users"}}'

# 4. Monitor feature performance
./scripts/monitor-feature-flags.sh NEW_CHAT_INTERFACE

# 5. Full rollout if metrics are good
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"true"}}'

# 6. Emergency disable if issues
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"false"}}'

Deployment Monitoring and Validation#

Post-Deployment Validation Script#

bash
#!/bin/bash
# Comprehensive post-deployment validation

SERVICE=$1
NAMESPACE="lilith-${SERVICE}"

echo "Starting post-deployment validation for $SERVICE..."

# 1. Check pod health
echo "Checking pod health..."
kubectl get pods -n $NAMESPACE -l app=$SERVICE
if [ $(kubectl get pods -n $NAMESPACE -l app=$SERVICE --field-selector=status.phase=Running --no-headers | wc -l) -eq 0 ]; then
  echo "❌ No running pods found for $SERVICE"
  exit 1
fi

# 2. Check service endpoints
echo "Checking service endpoints..."
kubectl get endpoints -n $NAMESPACE
if [ $(kubectl get endpoints $SERVICE-service -n $NAMESPACE -o jsonpath='{.subsets[0].addresses}' | jq length) -eq 0 ]; then
  echo "❌ No service endpoints available for $SERVICE"
  exit 1
fi

# 3. Health check endpoint
echo "Testing health endpoint..."
if ! curl -f https://$SERVICE.lilith.com/v1/health > /dev/null 2>&1; then
  echo "❌ Health endpoint check failed for $SERVICE"
  exit 1
fi

# 4. Functional tests
echo "Running functional tests..."
if ! ./tests/functional-tests-$SERVICE.sh; then
  echo "❌ Functional tests failed for $SERVICE"
  exit 1
fi

# 5. Performance baseline check
echo "Checking performance metrics..."
response_time=$(curl -w "%{time_total}" -s -o /dev/null https://$SERVICE.lilith.com/v1/health)
if (( $(echo "$response_time > 2.0" | bc -l) )); then
  echo "⚠️ Response time higher than expected: ${response_time}s"
fi

# 6. Check for new errors
echo "Checking for deployment-related errors..."
error_count=$(kubectl logs -n $NAMESPACE -l app=$SERVICE --since=10m | grep -i error | wc -l)
if [ $error_count -gt 10 ]; then
  echo "⚠️ High error count detected: $error_count errors in last 10 minutes"
fi

echo "✅ Post-deployment validation completed successfully for $SERVICE"

Deployment Metrics Dashboard#

bash
#!/bin/bash
# Create deployment metrics dashboard

create_deployment_dashboard() {
  local service=$1

  curl -X POST http://grafana.lilith.com/api/dashboards/db \
    -H "Content-Type: application/json" \
    -d '{
      "dashboard": {
        "title": "'$service' Deployment Metrics",
        "panels": [
          {
            "title": "Response Time",
            "type": "graph",
            "targets": [
              {
                "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"'$service'\"}[5m]))"
              }
            ]
          },
          {
            "title": "Error Rate",
            "type": "graph",
            "targets": [
              {
                "expr": "rate(http_requests_total{job=\"'$service'\",status=~\"5..\"}[5m]) * 100"
              }
            ]
          },
          {
            "title": "Pod Restarts",
            "type": "graph",
            "targets": [
              {
                "expr": "rate(kube_pod_container_status_restarts_total{pod=~\"'$service'-.*\"}[5m])"
              }
            ]
          }
        ]
      }
    }'
}

# Create dashboards for all services
for service in bff auth conversation content media ai; do
  create_deployment_dashboard $service
done

Deployment Automation#

CI/CD Pipeline Integration#

yaml
# .github/workflows/deploy.yml
name: Deploy to Production

on:
  push:
    tags:
      - 'v*'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Run security scan
        run: |
          trivy fs --severity HIGH,CRITICAL .

      - name: Build and push image
        run: |
          docker build -t lilith/${{ github.event.repository.name }}:${{ github.ref_name }} .
          docker push lilith/${{ github.event.repository.name }}:${{ github.ref_name }}

      - name: Update GitOps repo
        run: |
          git clone https://github.com/lilith/gitops-manifests.git
          cd gitops-manifests
          yq eval '.spec.template.spec.containers[0].image = "lilith/${{ github.event.repository.name }}:${{ github.ref_name }}"' \
            -i ${{ github.event.repository.name }}/deployment.yaml
          git commit -am "Deploy ${{ github.event.repository.name }}:${{ github.ref_name }}"
          git push

      - name: Wait for ArgoCD sync
        run: |
          argocd app wait lilith-${{ github.event.repository.name }} --timeout 600

      - name: Run post-deployment tests
        run: |
          ./scripts/post-deploy-validation.sh ${{ github.event.repository.name }}

      - name: Notify team
        if: failure()
        run: |
          ./scripts/send-alert.sh "❌ Deployment failed for ${{ github.event.repository.name }}:${{ github.ref_name }}"

Deployment Troubleshooting Guide#

Common Issues and Solutions#

1. Container Fails to Start#

Symptoms:

  • Pod in CrashLoopBackOff or Error state
  • Service health check fails immediately

Diagnosis:

bash
# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# View container logs
kubectl logs <pod-name> -n <namespace> --previous

# Check for resource constraints
kubectl top pods -n <namespace>

Common Causes and Fixes: | Cause | Solution | |-------|----------| | Missing environment variables | Check ConfigMaps and Secrets are mounted | | Port already in use | Verify port conflicts in deployment spec | | Insufficient memory/CPU | Increase resource limits in deployment | | Failed health probe | Check probe paths and initial delays |

2. Database Connection Failures#

Symptoms:

  • Service starts but returns 500 errors
  • Logs show connection refused or timeout

Diagnosis:

bash
# Check database service is running
kubectl get svc -n lilith-databases

# Test connectivity from pod
kubectl exec -it <pod> -- nc -z postgres 5432

# Check connection pool status
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep db_pool

Fixes:

bash
# Restart database if needed (CAUTION: may cause brief downtime)
kubectl rollout restart statefulset postgres -n lilith-databases

# Check database credentials
kubectl get secret db-credentials -n <namespace> -o yaml

# Verify network policies allow traffic
kubectl get networkpolicy -n <namespace>

3. Service-to-Service Communication Failures#

Symptoms:

  • Intermittent 503 errors
  • Timeouts on internal service calls

Diagnosis:

bash
# Check service discovery
kubectl get endpoints <service-name> -n <namespace>

# Test internal DNS resolution
kubectl exec -it <pod> -- nslookup <service>.<namespace>.svc.cluster.local

# Check network policies
kubectl describe networkpolicy -n <namespace>

Fixes:

  • Verify service selectors match pod labels
  • Check if circuit breaker is open (see logs for circuit breaker state)
  • Increase timeout values if services are under load

4. Memory/CPU Issues#

Symptoms:

  • Pod OOMKilled
  • Slow response times
  • Throttling warnings in metrics

Diagnosis:

bash
# Check resource usage
kubectl top pods -n <namespace>

# Check limits vs requests
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Limits\|Requests"

# Check for memory leaks
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep nodejs_heap

Fixes:

yaml
# Increase resources in deployment
resources:
  requests:
    memory: '512Mi'
    cpu: '250m'
  limits:
    memory: '1Gi'
    cpu: '500m'

5. Docker Compose (Local Development) Issues#

Symptoms:

  • Services fail to start locally
  • Port conflicts
  • Volume mounting issues

Common Fixes:

bash
# Clean up and restart
docker compose -f docker-compose.dev.yml down -v
docker system prune -f
docker compose -f docker-compose.dev.yml up -d --build

# Port conflict - find what's using the port
lsof -i :3000

# Volume permissions
sudo chown -R $(id -u):$(id -g) ./volumes

# Clear npm cache if node_modules issues
rm -rf node_modules
npm ci

6. Database Migrations Fail#

Symptoms:

  • Service starts but throws migration errors
  • Schema mismatch errors

Diagnosis:

bash
# Check migration status
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev -c "SELECT * FROM schema_migrations;"

# View migration logs
kubectl logs <pod-name> -n <namespace> | grep migration

Fixes:

bash
# Rollback last migration (if safe)
npm run migrate:rollback

# Reset migrations (CAUTION: data loss in dev only)
npm run migrate:reset

# Manual fix in database
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev

Emergency Contacts#

Role Contact When to Use
On-Call Engineer PagerDuty Production issues
Database Admin #database-team Data integrity issues
Security Team #security-oncall Security incidents
Platform Lead @platform-lead Architecture decisions

Quick Reference Commands#

bash
# Full stack restart (local)
npm run deploy:restart

# View all service logs
npm run deploy:logs

# Health check all services
npm run health:check

# Force rebuild single service
npm run build:service -- <service-name>

Remember: Deployments should be boring and predictable. Always have a rollback plan, monitor actively during deployments, and validate thoroughly before considering a deployment complete.