Eve library

Network Recovery Playbook

Runbook

Revision 1

On this page

Purpose: Network infrastructure failure recovery including DNS, load balancing, service mesh, and regional connectivity

Network Architecture Overview#

Network Topology#

text
Internet → CDN → Load Balancer → Ingress Controller → Service Mesh → Services
    ↓
Regional DNS → Geographic Load Balancing → Multi-AZ Deployment
    ↓
Primary Region (us-east-1) ← Backup Region (us-west-2)
    ↓                           ↓
  Kubernetes Cluster        Kubernetes Cluster
    ↓                           ↓
Service Mesh (Istio)      Service Mesh (Istio)
    ↓                           ↓
Internal Services         Internal Services

Critical Network Components#

text
External:
├── DNS (Route53/CloudFlare)
├── CDN (CloudFront/CloudFlare)
├── Load Balancer (ALB/NLB)
└── DDoS Protection (WAF)

Internal:
├── Kubernetes Networking (CNI)
├── Service Mesh (Istio/Linkerd)
├── Internal DNS (CoreDNS)
└── Pod-to-Pod Communication

Network Failure Scenarios#

DNS Failure Scenarios#

Scenario 1: Primary DNS Provider Failure#

text
Symptoms:
- Domain resolution failures
- Users unable to reach lilith.com
- DNS queries timing out
- CDN cannot resolve origin servers

DNS Recovery Process#

bash
# 1. Verify DNS failure scope
dig lilith.com @8.8.8.8
dig lilith.com @1.1.1.1
nslookup lilith.com

# 2. Check DNS provider status
curl -s https://status.route53.com/
curl -s https://www.cloudflarestatus.com/api/v2/status.json

# 3. Activate backup DNS provider
# Update DNS provider configuration
./scripts/dns-failover.sh --provider=backup --domain=lilith.com

# 4. Update DNS records to point to backup infrastructure
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://dns-backup-records.json

# 5. Verify DNS propagation
for server in 8.8.8.8 1.1.1.1 208.67.222.222; do
  echo "Checking DNS server $server:"
  dig @$server lilith.com +short
done

Geographic DNS Failover#

bash
# Implement geographic DNS failover
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "lilith.com",
        "Type": "A",
        "SetIdentifier": "Primary",
        "Failover": {"Type": "PRIMARY"},
        "TTL": 60,
        "ResourceRecords": [{"Value": "203.0.113.1"}]
      }
    }, {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "lilith.com",
        "Type": "A",
        "SetIdentifier": "Secondary",
        "Failover": {"Type": "SECONDARY"},
        "TTL": 60,
        "ResourceRecords": [{"Value": "203.0.113.2"}]
      }
    }]
  }'

Load Balancer Failure Recovery#

Scenario 2: Load Balancer Complete Failure#

text
Symptoms:
- 502/503 errors from load balancer
- Health checks failing
- Backend servers unreachable
- No traffic reaching Kubernetes ingress

Load Balancer Recovery#

bash
# 1. Check load balancer status
aws elbv2 describe-load-balancers --names lilith-main-alb
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:...

# 2. Check target health
kubectl get ingress -n lilith-bff
kubectl describe ingress main-ingress -n lilith-bff

# 3. Create new load balancer if needed
./scripts/create-emergency-lb.sh --cluster=lilith-prod --region=us-east-1

# 4. Update DNS to point to new load balancer
NEW_LB_DNS=$(aws elbv2 describe-load-balancers --names lilith-emergency-alb --query 'LoadBalancers[0].DNSName' --output text)

aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "lilith.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "'$NEW_LB_DNS'"}]
      }
    }]
  }'

# 5. Verify traffic flow
curl -v https://lilith.com/health

Service Mesh Failure Recovery#

Scenario 3: Service Mesh Control Plane Failure#

text
Symptoms:
- Inter-service communication failures
- mTLS certificate errors
- Service discovery not working
- Traffic routing failures

Service Mesh Recovery (Istio)#

bash
# 1. Check Istio control plane status
kubectl get pods -n istio-system
kubectl get svc -n istio-system

# 2. Check service mesh configuration
istioctl proxy-status
istioctl proxy-config cluster <pod-name> -n <namespace>

# 3. Restart Istio control plane components
kubectl rollout restart deployment istiod -n istio-system
kubectl rollout restart deployment istio-proxy -n istio-system

# 4. Regenerate service mesh certificates if needed
kubectl delete secret cacerts -n istio-system
kubectl create secret generic cacerts -n istio-system \
  --from-file=root-cert.pem \
  --from-file=cert-chain.pem \
  --from-file=ca-cert.pem \
  --from-file=ca-key.pem

# 5. Restart all service mesh sidecars
kubectl rollout restart deployment --all -n lilith-bff
kubectl rollout restart deployment --all -n lilith-auth
kubectl rollout restart deployment --all -n lilith-conversation

# 6. Verify service mesh connectivity
istioctl proxy-config cluster <pod-name> -n <namespace> | grep <service-name>

Regional Network Connectivity Failure#

Scenario 4: Complete Regional Network Isolation#

text
Symptoms:
- Cannot reach any services in primary region
- Network timeouts to all regional resources
- Cross-region database replication failures
- Monitoring systems show region offline

Regional Failover Process#

bash
# 1. Confirm regional isolation
ping $(dig +short us-east-1.compute.amazonaws.com)
traceroute primary-region.lilith.com

# 2. Activate disaster recovery region
kubectl config use-context lilith-dr-west-2

# 3. Scale up services in DR region
./scripts/activate-dr-region.sh --region=us-west-2

# 4. Update global load balancer to route to DR region
aws globalaccelerator update-endpoint-group \
  --endpoint-group-arn arn:aws:globalaccelerator::account:accelerator/... \
  --endpoint-configurations '[{
    "EndpointId": "arn:aws:elasticloadbalancing:us-west-2:account:loadbalancer/...",
    "Weight": 100
  }]'

# 5. Update DNS for geographic routing
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://geographic-failover.json

# 6. Verify DR region functionality
curl -v https://lilith.com/health
./scripts/e2e-test-dr-region.sh

Kubernetes Network Recovery#

CNI (Container Network Interface) Issues#

Pod Network Connectivity Failures#

bash
# 1. Check CNI plugin status
kubectl get nodes -o wide
kubectl describe nodes | grep -A5 "Network"

# 2. Check CNI daemonset
kubectl get daemonset -n kube-system
kubectl logs -n kube-system -l k8s-app=calico-node

# 3. Restart CNI components
kubectl rollout restart daemonset calico-node -n kube-system
kubectl rollout restart deployment calico-kube-controllers -n kube-system

# 4. Test pod-to-pod connectivity
kubectl run network-test-1 --image=busybox --rm -it -- sh
kubectl run network-test-2 --image=busybox --rm -it -- sh
# Inside pods: ping <other-pod-ip>

# 5. Check network policies
kubectl get networkpolicies --all-namespaces
kubectl describe networkpolicy <policy-name> -n <namespace>

CoreDNS Recovery#

Internal DNS Resolution Failures#

bash
# 1. Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 2. Check CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml

# 3. Test DNS resolution from pods
kubectl run dns-test --image=busybox --rm -it -- nslookup kubernetes.default.svc.cluster.local

# 4. Restart CoreDNS if needed
kubectl rollout restart deployment coredns -n kube-system

# 5. Flush DNS cache if resolution issues persist
kubectl exec -n kube-system deployment/coredns -- /bin/sh -c "killall -SIGUSR1 coredns"

CDN and Edge Recovery#

CDN Failure Recovery#

Scenario: CDN Provider Complete Outage#

bash
# 1. Check CDN status
curl -I https://cdn.lilith.com/
curl -s https://status.cloudflare.com/api/v2/status.json

# 2. Activate backup CDN provider
./scripts/cdn-failover.sh --provider=backup

# 3. Update DNS to point to backup CDN
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "cdn.lilith.com",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{"Value": "backup-cdn.example.com"}]
      }
    }]
  }'

# 4. Verify static asset delivery
curl -v https://cdn.lilith.com/assets/app.js
curl -v https://cdn.lilith.com/images/logo.png

# 5. Update application configuration if needed
kubectl patch configmap app-config -n lilith-bff \
  --patch='{"data":{"CDN_URL":"https://backup-cdn.example.com"}}'

Edge Location Recovery#

Geographic Edge Failures#

bash
# 1. Identify affected edge locations
aws cloudfront get-distribution --id EDGEIDENTIFIER

# 2. Update distribution to exclude failed edge locations
aws cloudfront update-distribution \
  --id EDGEIDENTIFIER \
  --if-match ETAG \
  --distribution-config file://updated-distribution.json

# 3. Monitor edge performance
./scripts/monitor-edge-performance.sh --region=all

# 4. Implement geographic routing backup
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://geographic-backup-routing.json

Network Security Recovery#

DDoS Attack Response#

DDoS Mitigation Procedures#

bash
# 1. Activate DDoS protection
aws shield-advanced create-protection \
  --name lilith-ddos-protection \
  --resource-arn arn:aws:elasticloadbalancing:region:account:loadbalancer/...

# 2. Enable WAF rate limiting
aws wafv2 update-web-acl \
  --scope CLOUDFRONT \
  --id WEB-ACL-ID \
  --default-action '{"Block": {}}'

# 3. Implement emergency rate limiting
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: rate-limit-emergency
  namespace: lilith-bff
spec:
  host: bff-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
EOF

# 4. Monitor attack patterns
./scripts/ddos-monitoring.sh --real-time

# 5. Coordinate with upstream providers
# Contact ISP/cloud provider for additional filtering

SSL/TLS Certificate Recovery#

Certificate Expiration or Compromise#

bash
# 1. Check certificate status
kubectl get certificates --all-namespaces
kubectl describe certificate lilith-tls -n lilith-bff

# 2. Generate new certificates
certbot certonly --dns-route53 -d lilith.com -d *.lilith.com

# 3. Update Kubernetes secrets
kubectl create secret tls lilith-tls-new \
  --cert=fullchain.pem \
  --key=privkey.pem \
  -n lilith-bff

# 4. Update ingress to use new certificates
kubectl patch ingress main-ingress -n lilith-bff \
  --patch='{"spec":{"tls":[{"secretName":"lilith-tls-new","hosts":["lilith.com","*.lilith.com"]}]}}'

# 5. Verify certificate deployment
curl -vI https://lilith.com/ 2>&1 | grep -A5 "Server certificate"
openssl s_client -connect lilith.com:443 -servername lilith.com < /dev/null 2>/dev/null | openssl x509 -noout -dates

Network Monitoring and Validation#

Network Health Validation Scripts#

Comprehensive Network Test#

bash
#!/bin/bash
# Comprehensive network connectivity test

test_external_connectivity() {
  echo "Testing external connectivity..."

  # Test DNS resolution
  if dig lilith.com +short > /dev/null; then
    echo "✓ DNS resolution working"
  else
    echo "✗ DNS resolution failed"
    return 1
  fi

  # Test HTTPS connectivity
  if curl -f https://lilith.com/health > /dev/null 2>&1; then
    echo "✓ HTTPS connectivity working"
  else
    echo "✗ HTTPS connectivity failed"
    return 1
  fi

  # Test CDN
  if curl -f https://cdn.lilith.com/health > /dev/null 2>&1; then
    echo "✓ CDN connectivity working"
  else
    echo "✗ CDN connectivity failed"
    return 1
  fi
}

test_internal_connectivity() {
  echo "Testing internal connectivity..."

  # Test pod-to-pod connectivity
  kubectl run network-test --image=busybox --rm -it --restart=Never -- \
    sh -c "nslookup kubernetes.default.svc.cluster.local && echo 'Internal DNS working'"

  # Test service mesh connectivity
  kubectl exec deployment/bff-deployment -n lilith-bff -- \
    curl -f http://auth-service.lilith-auth.svc.cluster.local/health
}

test_service_mesh() {
  echo "Testing service mesh..."

  # Check Istio proxy status
  istioctl proxy-status | grep -q SYNCED
  if [ $? -eq 0 ]; then
    echo "✓ Service mesh proxies synced"
  else
    echo "✗ Service mesh proxy sync issues"
    return 1
  fi

  # Test mTLS connectivity
  istioctl authn tls-check bff-deployment.lilith-bff.svc.cluster.local
}

# Run all tests
test_external_connectivity
test_internal_connectivity
test_service_mesh

echo "Network validation completed"

Real-time Network Monitoring#

bash
#!/bin/bash
# Real-time network monitoring script

monitor_network_health() {
  while true; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    # DNS resolution time
    dns_time=$(dig lilith.com | grep "Query time" | awk '{print $4}')

    # HTTPS response time
    https_time=$(curl -w "%{time_total}" -s -o /dev/null https://lilith.com/health)

    # CDN response time
    cdn_time=$(curl -w "%{time_total}" -s -o /dev/null https://cdn.lilith.com/health)

    echo "$timestamp,DNS:${dns_time}ms,HTTPS:${https_time}s,CDN:${cdn_time}s"

    # Alert if response times are too high
    if (( $(echo "$https_time > 5.0" | bc -l) )); then
      ./scripts/send-alert.sh "High HTTPS response time: ${https_time}s"
    fi

    sleep 30
  done
}

monitor_network_health

Automated Network Recovery#

Self-Healing Network Scripts#

bash
#!/bin/bash
# Automated network recovery script

auto_recover_network() {
  local component=$1

  case $component in
    "dns")
      # DNS auto-recovery
      if ! dig lilith.com +short > /dev/null 2>&1; then
        echo "DNS failure detected, initiating auto-recovery..."
        ./scripts/dns-failover.sh --auto

        # Wait and verify
        sleep 60
        if dig lilith.com +short > /dev/null 2>&1; then
          echo "DNS auto-recovery successful"
          ./scripts/send-alert.sh "✓ DNS auto-recovery successful"
        else
          echo "DNS auto-recovery failed, escalating..."
          ./scripts/escalate-incident.sh "DNS auto-recovery failed"
        fi
      fi
      ;;

    "service-mesh")
      # Service mesh auto-recovery
      if ! istioctl proxy-status | grep -q SYNCED; then
        echo "Service mesh failure detected, restarting control plane..."
        kubectl rollout restart deployment istiod -n istio-system

        # Wait for recovery
        kubectl wait --for=condition=Available deployment istiod -n istio-system --timeout=300s

        # Verify recovery
        if istioctl proxy-status | grep -q SYNCED; then
          echo "Service mesh auto-recovery successful"
        else
          echo "Service mesh auto-recovery failed, escalating..."
          ./scripts/escalate-incident.sh "Service mesh auto-recovery failed"
        fi
      fi
      ;;

    "ingress")
      # Ingress controller auto-recovery
      if ! curl -f https://lilith.com/health > /dev/null 2>&1; then
        echo "Ingress failure detected, restarting ingress controller..."
        kubectl rollout restart deployment nginx-ingress-controller -n ingress-nginx

        # Wait and verify
        kubectl wait --for=condition=Available deployment nginx-ingress-controller -n ingress-nginx --timeout=300s

        if curl -f https://lilith.com/health > /dev/null 2>&1; then
          echo "Ingress auto-recovery successful"
        else
          echo "Ingress auto-recovery failed, escalating..."
          ./scripts/escalate-incident.sh "Ingress auto-recovery failed"
        fi
      fi
      ;;
  esac
}

# Monitor and auto-recover all network components
while true; do
  auto_recover_network "dns"
  auto_recover_network "service-mesh"
  auto_recover_network "ingress"
  sleep 60
done

Remember: Network recovery often requires coordination across multiple layers. Always verify connectivity at each layer (DNS → Load Balancer → Ingress → Service Mesh → Services) and maintain redundancy at every level.