Eve library

Infrastructure Alert Runbooks

Runbook

Revision 1

On this page

This document provides runbooks for Kubernetes and infrastructure-related alerts in the Lilith platform.

Overview#

Infrastructure alerts are triggered by the kube-prometheus-stack and monitor the health of Kubernetes clusters, nodes, pods, and core infrastructure components.

Alert Index#

Alert Severity Description
KubePodCrashLooping Critical Pod restarting repeatedly
KubePodNotReady Warning Pod not in ready state
KubeDeploymentReplicasMismatch Warning Deployment has wrong replica count
KubeNodeNotReady Critical Node not in ready state
KubeNodeUnreachable Critical Node unreachable
KubeCPUOvercommit Warning CPU overcommitted on cluster
KubeMemoryOvercommit Warning Memory overcommitted on cluster
NodeDiskRunningFull Critical Disk space running low
NodeHighCPUUsage Warning High CPU utilization
NodeHighMemoryUsage Warning High memory utilization

KubePodCrashLooping#

Description#

A pod is restarting repeatedly (crash loop). This typically indicates an application issue, misconfiguration, or resource constraint.

Severity#

Critical - Immediate attention required

Trigger Condition#

promql
rate(kube_pod_container_status_restarts_total{namespace=~"lilith.*"}[15m]) * 60 * 5 > 0

Impact#

  • Service degradation or outage
  • Potential data inconsistency
  • User-facing errors

Investigation Steps#

  1. Identify the crashing pod

    bash
    kubectl get pods -n <namespace> | grep -E "(CrashLoopBackOff|Error)"
    
  2. Check pod events

    bash
    kubectl describe pod <pod-name> -n <namespace>
    
  3. Check logs from previous container

    bash
    kubectl logs <pod-name> -n <namespace> --previous
    kubectl logs <pod-name> -n <namespace> --tail=100
    
  4. Check resource usage

    bash
    kubectl top pod <pod-name> -n <namespace>
    
  5. Check for OOMKilled

    bash
    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
    

Remediation Actions#

  1. If OOMKilled:

    bash
    # Increase memory limits
    kubectl patch deployment <deployment> -n <namespace> \
      -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"1Gi"}}}]}}}}'
    
  2. If configuration error:

    • Check ConfigMaps and Secrets
    • Verify environment variables
    • Review startup command
  3. If dependency unavailable:

    • Check dependent services
    • Verify database connectivity
    • Check network policies
  4. If code bug:

    • Roll back to last known good version
    • Escalate to development team

Escalation Path#

  1. On-call SRE
  2. Service team (if application-related)
  3. Platform team (if infrastructure-related)

KubePodNotReady#

Description#

A pod is running but not in the Ready state. This means the readiness probe is failing.

Severity#

Warning - Investigate within 15 minutes

Trigger Condition#

promql
kube_pod_status_ready{condition="false", namespace=~"lilith.*"} == 1

Investigation Steps#

  1. Check pod status

    bash
    kubectl get pods -n <namespace> -o wide
    kubectl describe pod <pod-name> -n <namespace>
    
  2. Check readiness probe configuration

    bash
    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}'
    
  3. Test readiness endpoint manually

    bash
    kubectl exec -it <pod-name> -n <namespace> -- curl localhost:<port>/health/ready
    
  4. Check logs for errors

    bash
    kubectl logs <pod-name> -n <namespace> --tail=50
    

Remediation Actions#

  1. If probe failing due to dependency:

    • Check dependent services
    • Consider adjusting probe settings
  2. If probe misconfigured:

    • Review probe path and port
    • Adjust initialDelaySeconds if startup is slow
  3. If resource-related:

    • Check CPU/memory utilization
    • Scale resources if needed

KubeDeploymentReplicasMismatch#

Description#

The deployment has fewer replicas than desired. Pods may be pending, failed, or unable to schedule.

Severity#

Warning - Investigate within 30 minutes

Trigger Condition#

promql
kube_deployment_status_replicas_available != kube_deployment_spec_replicas

Investigation Steps#

  1. Check deployment status

    bash
    kubectl get deployment <deployment> -n <namespace>
    kubectl describe deployment <deployment> -n <namespace>
    
  2. Check for pending pods

    bash
    kubectl get pods -n <namespace> | grep Pending
    kubectl describe pod <pending-pod> -n <namespace>
    
  3. Check for scheduling issues

    bash
    kubectl get events -n <namespace> --field-selector reason=FailedScheduling
    
  4. Check node resources

    bash
    kubectl describe nodes | grep -A5 "Allocated resources"
    

Remediation Actions#

  1. If scheduling failure due to resources:

    bash
    # Scale up node group
    # Or reduce resource requests
    kubectl patch deployment <deployment> -n <namespace> \
      -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"memory":"256Mi","cpu":"100m"}}}]}}}}'
    
  2. If node affinity issue:

    • Review node affinity rules
    • Add appropriate node labels
    • Consider removing constraints
  3. If PVC issues:

    • Check PersistentVolumeClaim status
    • Verify storage class availability

KubeNodeNotReady#

Description#

A Kubernetes node is not in the Ready state. Pods on this node may be affected.

Severity#

Critical - Immediate attention required

Trigger Condition#

promql
kube_node_status_condition{condition="Ready",status="true"} == 0

Investigation Steps#

  1. Check node status

    bash
    kubectl get nodes
    kubectl describe node <node-name>
    
  2. Check node conditions

    bash
    kubectl get node <node-name> -o jsonpath='{.status.conditions[*]}'
    
  3. Check kubelet status (if accessible)

    bash
    ssh <node> systemctl status kubelet
    ssh <node> journalctl -u kubelet --since="10 minutes ago"
    
  4. Check for resource pressure

    bash
    kubectl describe node <node-name> | grep -A10 "Conditions:"
    

Remediation Actions#

  1. If node is recoverable:

    bash
    # Restart kubelet
    ssh <node> sudo systemctl restart kubelet
    
  2. If node is unrecoverable:

    bash
    # Cordon and drain
    kubectl cordon <node-name>
    kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
    
    # Replace node (EKS/GKE/AKS specific)
    
  3. If disk pressure:

    • Clean up unused images
    • Expand disk or clean logs
  4. If memory pressure:

    • Evict low-priority pods
    • Add capacity to cluster

Escalation Path#

  1. On-call SRE
  2. Platform team
  3. Cloud provider support (if hardware issue)

KubeNodeUnreachable#

Description#

A Kubernetes node cannot be reached by the control plane.

Severity#

Critical - Immediate attention required

Investigation Steps#

  1. Check node status

    bash
    kubectl get nodes
    
  2. Check cloud provider console

    • Verify instance is running
    • Check network connectivity
    • Review security groups
  3. Check VPC and networking

    • Verify VPC routes
    • Check NAT gateway status
    • Review network ACLs

Remediation Actions#

  1. If network issue:

    • Check security group rules
    • Verify VPC configuration
    • Review network policies
  2. If instance crashed:

    • Attempt instance restart via cloud console
    • If unrecoverable, terminate and replace

KubeCPUOvercommit#

Description#

The cluster has overcommitted CPU - sum of CPU requests exceeds available capacity.

Severity#

Warning - Plan capacity increase

Trigger Condition#

promql
sum(kube_pod_container_resource_requests{resource="cpu"})
/
sum(kube_node_status_allocatable{resource="cpu"}) > 1

Investigation Steps#

  1. Check cluster CPU allocation

    bash
    kubectl describe nodes | grep -A10 "Allocated resources"
    
  2. Find highest CPU consumers

    bash
    kubectl top pods --all-namespaces --sort-by=cpu | head -20
    

Remediation Actions#

  1. Add cluster capacity:

    • Scale up node group
    • Add new node pool
  2. Optimize requests:

    • Review pod CPU requests
    • Right-size based on actual usage

KubeMemoryOvercommit#

Description#

The cluster has overcommitted memory - sum of memory requests exceeds available capacity.

Severity#

Warning - Plan capacity increase

Investigation Steps#

  1. Check cluster memory allocation

    bash
    kubectl describe nodes | grep -A10 "Allocated resources"
    
  2. Find highest memory consumers

    bash
    kubectl top pods --all-namespaces --sort-by=memory | head -20
    

Remediation Actions#

Same as CPU overcommit - add capacity or optimize requests.


NodeDiskRunningFull#

Description#

Node disk space is running low (< 15% free).

Severity#

Critical - Immediate attention required

Trigger Condition#

promql
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15

Investigation Steps#

  1. Check disk usage

    bash
    df -h
    du -sh /* | sort -hr | head -20
    
  2. Check for large log files

    bash
    find /var/log -type f -size +100M
    
  3. Check container images

    bash
    docker system df
    crictl images | wc -l
    

Remediation Actions#

  1. Clean up Docker/containerd images

    bash
    docker system prune -a
    crictl rmi --prune
    
  2. Clean up old logs

    bash
    journalctl --vacuum-time=7d
    find /var/log -type f -name "*.gz" -mtime +7 -delete
    
  3. Expand disk (cloud-specific)

    • EKS: Modify EBS volume, then resize filesystem
    • GKE: Edit node pool disk size
    • AKS: Scale VM disk

NodeHighCPUUsage#

Description#

Node CPU utilization is consistently high (> 90%).

Severity#

Warning - Investigate and plan capacity

Trigger Condition#

promql
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90

Investigation Steps#

  1. Identify high CPU processes

    bash
    kubectl top pods -A --sort-by=cpu | head -20
    
  2. Check for CPU throttling

    promql
    rate(container_cpu_cfs_throttled_seconds_total[5m])
    

Remediation Actions#

  1. Scale out workloads to distribute load
  2. Increase node size if consistently overloaded
  3. Optimize application if specific service is the cause

NodeHighMemoryUsage#

Description#

Node memory utilization is high (> 90%).

Severity#

Warning - Risk of OOM conditions

Trigger Condition#

promql
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90

Investigation Steps#

  1. Identify high memory pods

    bash
    kubectl top pods -A --sort-by=memory | head -20
    
  2. Check for memory leaks

    promql
    container_memory_usage_bytes{namespace=~"lilith.*"}[1h]
    

Remediation Actions#

  1. Scale or restart memory-leaking pods
  2. Increase node size
  3. Add memory limits to prevent runaway processes

General Commands#

bash
# Cluster overview
kubectl get nodes -o wide
kubectl top nodes

# Pod status across namespaces
kubectl get pods -A | grep -v Running

# Recent events
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

# Resource allocation
kubectl describe nodes | grep -A20 "Allocated resources"

Contact Information#

Role Contact
On-Call SRE #sre-oncall (Slack)
Platform Team #platform-team (Slack)
Cloud Team #cloud-infrastructure (Slack)

References#