Purpose: Network infrastructure failure recovery including DNS, load balancing, service mesh, and regional connectivity
Network Architecture Overview#
Network Topology#
text
Internet → CDN → Load Balancer → Ingress Controller → Service Mesh → Services
↓
Regional DNS → Geographic Load Balancing → Multi-AZ Deployment
↓
Primary Region (us-east-1) ← Backup Region (us-west-2)
↓ ↓
Kubernetes Cluster Kubernetes Cluster
↓ ↓
Service Mesh (Istio) Service Mesh (Istio)
↓ ↓
Internal Services Internal Services
Critical Network Components#
text
External:
├── DNS (Route53/CloudFlare)
├── CDN (CloudFront/CloudFlare)
├── Load Balancer (ALB/NLB)
└── DDoS Protection (WAF)
Internal:
├── Kubernetes Networking (CNI)
├── Service Mesh (Istio/Linkerd)
├── Internal DNS (CoreDNS)
└── Pod-to-Pod Communication
Network Failure Scenarios#
DNS Failure Scenarios#
Scenario 1: Primary DNS Provider Failure#
text
Symptoms:
- Domain resolution failures
- Users unable to reach lilith.com
- DNS queries timing out
- CDN cannot resolve origin servers
DNS Recovery Process#
bash
# 1. Verify DNS failure scope
dig lilith.com @8.8.8.8
dig lilith.com @1.1.1.1
nslookup lilith.com
# 2. Check DNS provider status
curl -s https://status.route53.com/
curl -s https://www.cloudflarestatus.com/api/v2/status.json
# 3. Activate backup DNS provider
# Update DNS provider configuration
./scripts/dns-failover.sh --provider=backup --domain=lilith.com
# 4. Update DNS records to point to backup infrastructure
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch file://dns-backup-records.json
# 5. Verify DNS propagation
for server in 8.8.8.8 1.1.1.1 208.67.222.222; do
echo "Checking DNS server $server:"
dig @$server lilith.com +short
done
Geographic DNS Failover#
bash
# Implement geographic DNS failover
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch '{
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "lilith.com",
"Type": "A",
"SetIdentifier": "Primary",
"Failover": {"Type": "PRIMARY"},
"TTL": 60,
"ResourceRecords": [{"Value": "203.0.113.1"}]
}
}, {
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "lilith.com",
"Type": "A",
"SetIdentifier": "Secondary",
"Failover": {"Type": "SECONDARY"},
"TTL": 60,
"ResourceRecords": [{"Value": "203.0.113.2"}]
}
}]
}'
Load Balancer Failure Recovery#
Scenario 2: Load Balancer Complete Failure#
text
Symptoms:
- 502/503 errors from load balancer
- Health checks failing
- Backend servers unreachable
- No traffic reaching Kubernetes ingress
Load Balancer Recovery#
bash
# 1. Check load balancer status
aws elbv2 describe-load-balancers --names lilith-main-alb
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:...
# 2. Check target health
kubectl get ingress -n lilith-bff
kubectl describe ingress main-ingress -n lilith-bff
# 3. Create new load balancer if needed
./scripts/create-emergency-lb.sh --cluster=lilith-prod --region=us-east-1
# 4. Update DNS to point to new load balancer
NEW_LB_DNS=$(aws elbv2 describe-load-balancers --names lilith-emergency-alb --query 'LoadBalancers[0].DNSName' --output text)
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch '{
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "lilith.com",
"Type": "CNAME",
"TTL": 60,
"ResourceRecords": [{"Value": "'$NEW_LB_DNS'"}]
}
}]
}'
# 5. Verify traffic flow
curl -v https://lilith.com/health
Service Mesh Failure Recovery#
Scenario 3: Service Mesh Control Plane Failure#
text
Symptoms:
- Inter-service communication failures
- mTLS certificate errors
- Service discovery not working
- Traffic routing failures
Service Mesh Recovery (Istio)#
bash
# 1. Check Istio control plane status
kubectl get pods -n istio-system
kubectl get svc -n istio-system
# 2. Check service mesh configuration
istioctl proxy-status
istioctl proxy-config cluster <pod-name> -n <namespace>
# 3. Restart Istio control plane components
kubectl rollout restart deployment istiod -n istio-system
kubectl rollout restart deployment istio-proxy -n istio-system
# 4. Regenerate service mesh certificates if needed
kubectl delete secret cacerts -n istio-system
kubectl create secret generic cacerts -n istio-system \
--from-file=root-cert.pem \
--from-file=cert-chain.pem \
--from-file=ca-cert.pem \
--from-file=ca-key.pem
# 5. Restart all service mesh sidecars
kubectl rollout restart deployment --all -n lilith-bff
kubectl rollout restart deployment --all -n lilith-auth
kubectl rollout restart deployment --all -n lilith-conversation
# 6. Verify service mesh connectivity
istioctl proxy-config cluster <pod-name> -n <namespace> | grep <service-name>
Regional Network Connectivity Failure#
Scenario 4: Complete Regional Network Isolation#
text
Symptoms:
- Cannot reach any services in primary region
- Network timeouts to all regional resources
- Cross-region database replication failures
- Monitoring systems show region offline
Regional Failover Process#
bash
# 1. Confirm regional isolation
ping $(dig +short us-east-1.compute.amazonaws.com)
traceroute primary-region.lilith.com
# 2. Activate disaster recovery region
kubectl config use-context lilith-dr-west-2
# 3. Scale up services in DR region
./scripts/activate-dr-region.sh --region=us-west-2
# 4. Update global load balancer to route to DR region
aws globalaccelerator update-endpoint-group \
--endpoint-group-arn arn:aws:globalaccelerator::account:accelerator/... \
--endpoint-configurations '[{
"EndpointId": "arn:aws:elasticloadbalancing:us-west-2:account:loadbalancer/...",
"Weight": 100
}]'
# 5. Update DNS for geographic routing
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch file://geographic-failover.json
# 6. Verify DR region functionality
curl -v https://lilith.com/health
./scripts/e2e-test-dr-region.sh
Kubernetes Network Recovery#
CNI (Container Network Interface) Issues#
Pod Network Connectivity Failures#
bash
# 1. Check CNI plugin status
kubectl get nodes -o wide
kubectl describe nodes | grep -A5 "Network"
# 2. Check CNI daemonset
kubectl get daemonset -n kube-system
kubectl logs -n kube-system -l k8s-app=calico-node
# 3. Restart CNI components
kubectl rollout restart daemonset calico-node -n kube-system
kubectl rollout restart deployment calico-kube-controllers -n kube-system
# 4. Test pod-to-pod connectivity
kubectl run network-test-1 --image=busybox --rm -it -- sh
kubectl run network-test-2 --image=busybox --rm -it -- sh
# Inside pods: ping <other-pod-ip>
# 5. Check network policies
kubectl get networkpolicies --all-namespaces
kubectl describe networkpolicy <policy-name> -n <namespace>
CoreDNS Recovery#
Internal DNS Resolution Failures#
bash
# 1. Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# 2. Check CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml
# 3. Test DNS resolution from pods
kubectl run dns-test --image=busybox --rm -it -- nslookup kubernetes.default.svc.cluster.local
# 4. Restart CoreDNS if needed
kubectl rollout restart deployment coredns -n kube-system
# 5. Flush DNS cache if resolution issues persist
kubectl exec -n kube-system deployment/coredns -- /bin/sh -c "killall -SIGUSR1 coredns"
CDN and Edge Recovery#
CDN Failure Recovery#
Scenario: CDN Provider Complete Outage#
bash
# 1. Check CDN status
curl -I https://cdn.lilith.com/
curl -s https://status.cloudflare.com/api/v2/status.json
# 2. Activate backup CDN provider
./scripts/cdn-failover.sh --provider=backup
# 3. Update DNS to point to backup CDN
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch '{
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "cdn.lilith.com",
"Type": "CNAME",
"TTL": 300,
"ResourceRecords": [{"Value": "backup-cdn.example.com"}]
}
}]
}'
# 4. Verify static asset delivery
curl -v https://cdn.lilith.com/assets/app.js
curl -v https://cdn.lilith.com/images/logo.png
# 5. Update application configuration if needed
kubectl patch configmap app-config -n lilith-bff \
--patch='{"data":{"CDN_URL":"https://backup-cdn.example.com"}}'
Edge Location Recovery#
Geographic Edge Failures#
bash
# 1. Identify affected edge locations
aws cloudfront get-distribution --id EDGEIDENTIFIER
# 2. Update distribution to exclude failed edge locations
aws cloudfront update-distribution \
--id EDGEIDENTIFIER \
--if-match ETAG \
--distribution-config file://updated-distribution.json
# 3. Monitor edge performance
./scripts/monitor-edge-performance.sh --region=all
# 4. Implement geographic routing backup
aws route53 change-resource-record-sets \
--hosted-zone-id Z123456789 \
--change-batch file://geographic-backup-routing.json
Network Security Recovery#
DDoS Attack Response#
DDoS Mitigation Procedures#
bash
# 1. Activate DDoS protection
aws shield-advanced create-protection \
--name lilith-ddos-protection \
--resource-arn arn:aws:elasticloadbalancing:region:account:loadbalancer/...
# 2. Enable WAF rate limiting
aws wafv2 update-web-acl \
--scope CLOUDFRONT \
--id WEB-ACL-ID \
--default-action '{"Block": {}}'
# 3. Implement emergency rate limiting
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: rate-limit-emergency
namespace: lilith-bff
spec:
host: bff-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http1MaxPendingRequests: 50
maxRequestsPerConnection: 10
EOF
# 4. Monitor attack patterns
./scripts/ddos-monitoring.sh --real-time
# 5. Coordinate with upstream providers
# Contact ISP/cloud provider for additional filtering
SSL/TLS Certificate Recovery#
Certificate Expiration or Compromise#
bash
# 1. Check certificate status
kubectl get certificates --all-namespaces
kubectl describe certificate lilith-tls -n lilith-bff
# 2. Generate new certificates
certbot certonly --dns-route53 -d lilith.com -d *.lilith.com
# 3. Update Kubernetes secrets
kubectl create secret tls lilith-tls-new \
--cert=fullchain.pem \
--key=privkey.pem \
-n lilith-bff
# 4. Update ingress to use new certificates
kubectl patch ingress main-ingress -n lilith-bff \
--patch='{"spec":{"tls":[{"secretName":"lilith-tls-new","hosts":["lilith.com","*.lilith.com"]}]}}'
# 5. Verify certificate deployment
curl -vI https://lilith.com/ 2>&1 | grep -A5 "Server certificate"
openssl s_client -connect lilith.com:443 -servername lilith.com < /dev/null 2>/dev/null | openssl x509 -noout -dates
Network Monitoring and Validation#
Network Health Validation Scripts#
Comprehensive Network Test#
bash
#!/bin/bash
# Comprehensive network connectivity test
test_external_connectivity() {
echo "Testing external connectivity..."
# Test DNS resolution
if dig lilith.com +short > /dev/null; then
echo "✓ DNS resolution working"
else
echo "✗ DNS resolution failed"
return 1
fi
# Test HTTPS connectivity
if curl -f https://lilith.com/health > /dev/null 2>&1; then
echo "✓ HTTPS connectivity working"
else
echo "✗ HTTPS connectivity failed"
return 1
fi
# Test CDN
if curl -f https://cdn.lilith.com/health > /dev/null 2>&1; then
echo "✓ CDN connectivity working"
else
echo "✗ CDN connectivity failed"
return 1
fi
}
test_internal_connectivity() {
echo "Testing internal connectivity..."
# Test pod-to-pod connectivity
kubectl run network-test --image=busybox --rm -it --restart=Never -- \
sh -c "nslookup kubernetes.default.svc.cluster.local && echo 'Internal DNS working'"
# Test service mesh connectivity
kubectl exec deployment/bff-deployment -n lilith-bff -- \
curl -f http://auth-service.lilith-auth.svc.cluster.local/health
}
test_service_mesh() {
echo "Testing service mesh..."
# Check Istio proxy status
istioctl proxy-status | grep -q SYNCED
if [ $? -eq 0 ]; then
echo "✓ Service mesh proxies synced"
else
echo "✗ Service mesh proxy sync issues"
return 1
fi
# Test mTLS connectivity
istioctl authn tls-check bff-deployment.lilith-bff.svc.cluster.local
}
# Run all tests
test_external_connectivity
test_internal_connectivity
test_service_mesh
echo "Network validation completed"
Real-time Network Monitoring#
bash
#!/bin/bash
# Real-time network monitoring script
monitor_network_health() {
while true; do
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
# DNS resolution time
dns_time=$(dig lilith.com | grep "Query time" | awk '{print $4}')
# HTTPS response time
https_time=$(curl -w "%{time_total}" -s -o /dev/null https://lilith.com/health)
# CDN response time
cdn_time=$(curl -w "%{time_total}" -s -o /dev/null https://cdn.lilith.com/health)
echo "$timestamp,DNS:${dns_time}ms,HTTPS:${https_time}s,CDN:${cdn_time}s"
# Alert if response times are too high
if (( $(echo "$https_time > 5.0" | bc -l) )); then
./scripts/send-alert.sh "High HTTPS response time: ${https_time}s"
fi
sleep 30
done
}
monitor_network_health
Automated Network Recovery#
Self-Healing Network Scripts#
bash
#!/bin/bash
# Automated network recovery script
auto_recover_network() {
local component=$1
case $component in
"dns")
# DNS auto-recovery
if ! dig lilith.com +short > /dev/null 2>&1; then
echo "DNS failure detected, initiating auto-recovery..."
./scripts/dns-failover.sh --auto
# Wait and verify
sleep 60
if dig lilith.com +short > /dev/null 2>&1; then
echo "DNS auto-recovery successful"
./scripts/send-alert.sh "✓ DNS auto-recovery successful"
else
echo "DNS auto-recovery failed, escalating..."
./scripts/escalate-incident.sh "DNS auto-recovery failed"
fi
fi
;;
"service-mesh")
# Service mesh auto-recovery
if ! istioctl proxy-status | grep -q SYNCED; then
echo "Service mesh failure detected, restarting control plane..."
kubectl rollout restart deployment istiod -n istio-system
# Wait for recovery
kubectl wait --for=condition=Available deployment istiod -n istio-system --timeout=300s
# Verify recovery
if istioctl proxy-status | grep -q SYNCED; then
echo "Service mesh auto-recovery successful"
else
echo "Service mesh auto-recovery failed, escalating..."
./scripts/escalate-incident.sh "Service mesh auto-recovery failed"
fi
fi
;;
"ingress")
# Ingress controller auto-recovery
if ! curl -f https://lilith.com/health > /dev/null 2>&1; then
echo "Ingress failure detected, restarting ingress controller..."
kubectl rollout restart deployment nginx-ingress-controller -n ingress-nginx
# Wait and verify
kubectl wait --for=condition=Available deployment nginx-ingress-controller -n ingress-nginx --timeout=300s
if curl -f https://lilith.com/health > /dev/null 2>&1; then
echo "Ingress auto-recovery successful"
else
echo "Ingress auto-recovery failed, escalating..."
./scripts/escalate-incident.sh "Ingress auto-recovery failed"
fi
fi
;;
esac
}
# Monitor and auto-recover all network components
while true; do
auto_recover_network "dns"
auto_recover_network "service-mesh"
auto_recover_network "ingress"
sleep 60
done
Remember: Network recovery often requires coordination across multiple layers. Always verify connectivity at each layer (DNS → Load Balancer → Ingress → Service Mesh → Services) and maintain redundancy at every level.