# Infrastructure Alert Runbooks

This document provides runbooks for Kubernetes and infrastructure-related alerts
in the Lilith platform.

## Overview

Infrastructure alerts are triggered by the kube-prometheus-stack and monitor the
health of Kubernetes clusters, nodes, pods, and core infrastructure components.

## Alert Index

| Alert                                                             | Severity | Description                        |
| ----------------------------------------------------------------- | -------- | ---------------------------------- |
| [KubePodCrashLooping](#kubepodcrashlooping)                       | Critical | Pod restarting repeatedly          |
| [KubePodNotReady](#kubepodnotready)                               | Warning  | Pod not in ready state             |
| [KubeDeploymentReplicasMismatch](#kubedeploymentreplicasmismatch) | Warning  | Deployment has wrong replica count |
| [KubeNodeNotReady](#kubenodenotready)                             | Critical | Node not in ready state            |
| [KubeNodeUnreachable](#kubenodeunreachable)                       | Critical | Node unreachable                   |
| [KubeCPUOvercommit](#kubecpuovercommit)                           | Warning  | CPU overcommitted on cluster       |
| [KubeMemoryOvercommit](#kubememoryovercommit)                     | Warning  | Memory overcommitted on cluster    |
| [NodeDiskRunningFull](#nodediskrunningfull)                       | Critical | Disk space running low             |
| [NodeHighCPUUsage](#nodehighcpuusage)                             | Warning  | High CPU utilization               |
| [NodeHighMemoryUsage](#nodehighmemoryusage)                       | Warning  | High memory utilization            |

---

## KubePodCrashLooping

### Description

A pod is restarting repeatedly (crash loop). This typically indicates an
application issue, misconfiguration, or resource constraint.

### Severity

**Critical** - Immediate attention required

### Trigger Condition

```promql
rate(kube_pod_container_status_restarts_total{namespace=~"lilith.*"}[15m]) * 60 * 5 > 0
```

### Impact

- Service degradation or outage
- Potential data inconsistency
- User-facing errors

### Investigation Steps

1. **Identify the crashing pod**

   ```bash
   kubectl get pods -n <namespace> | grep -E "(CrashLoopBackOff|Error)"
   ```

2. **Check pod events**

   ```bash
   kubectl describe pod <pod-name> -n <namespace>
   ```

3. **Check logs from previous container**

   ```bash
   kubectl logs <pod-name> -n <namespace> --previous
   kubectl logs <pod-name> -n <namespace> --tail=100
   ```

4. **Check resource usage**

   ```bash
   kubectl top pod <pod-name> -n <namespace>
   ```

5. **Check for OOMKilled**
   ```bash
   kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
   ```

### Remediation Actions

1. **If OOMKilled:**

   ```bash
   # Increase memory limits
   kubectl patch deployment <deployment> -n <namespace> \
     -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"1Gi"}}}]}}}}'
   ```

2. **If configuration error:**
   - Check ConfigMaps and Secrets
   - Verify environment variables
   - Review startup command

3. **If dependency unavailable:**
   - Check dependent services
   - Verify database connectivity
   - Check network policies

4. **If code bug:**
   - Roll back to last known good version
   - Escalate to development team

### Escalation Path

1. On-call SRE
2. Service team (if application-related)
3. Platform team (if infrastructure-related)

---

## KubePodNotReady

### Description

A pod is running but not in the Ready state. This means the readiness probe is
failing.

### Severity

**Warning** - Investigate within 15 minutes

### Trigger Condition

```promql
kube_pod_status_ready{condition="false", namespace=~"lilith.*"} == 1
```

### Investigation Steps

1. **Check pod status**

   ```bash
   kubectl get pods -n <namespace> -o wide
   kubectl describe pod <pod-name> -n <namespace>
   ```

2. **Check readiness probe configuration**

   ```bash
   kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}'
   ```

3. **Test readiness endpoint manually**

   ```bash
   kubectl exec -it <pod-name> -n <namespace> -- curl localhost:<port>/health/ready
   ```

4. **Check logs for errors**
   ```bash
   kubectl logs <pod-name> -n <namespace> --tail=50
   ```

### Remediation Actions

1. **If probe failing due to dependency:**
   - Check dependent services
   - Consider adjusting probe settings

2. **If probe misconfigured:**
   - Review probe path and port
   - Adjust initialDelaySeconds if startup is slow

3. **If resource-related:**
   - Check CPU/memory utilization
   - Scale resources if needed

---

## KubeDeploymentReplicasMismatch

### Description

The deployment has fewer replicas than desired. Pods may be pending, failed, or
unable to schedule.

### Severity

**Warning** - Investigate within 30 minutes

### Trigger Condition

```promql
kube_deployment_status_replicas_available != kube_deployment_spec_replicas
```

### Investigation Steps

1. **Check deployment status**

   ```bash
   kubectl get deployment <deployment> -n <namespace>
   kubectl describe deployment <deployment> -n <namespace>
   ```

2. **Check for pending pods**

   ```bash
   kubectl get pods -n <namespace> | grep Pending
   kubectl describe pod <pending-pod> -n <namespace>
   ```

3. **Check for scheduling issues**

   ```bash
   kubectl get events -n <namespace> --field-selector reason=FailedScheduling
   ```

4. **Check node resources**
   ```bash
   kubectl describe nodes | grep -A5 "Allocated resources"
   ```

### Remediation Actions

1. **If scheduling failure due to resources:**

   ```bash
   # Scale up node group
   # Or reduce resource requests
   kubectl patch deployment <deployment> -n <namespace> \
     -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"memory":"256Mi","cpu":"100m"}}}]}}}}'
   ```

2. **If node affinity issue:**
   - Review node affinity rules
   - Add appropriate node labels
   - Consider removing constraints

3. **If PVC issues:**
   - Check PersistentVolumeClaim status
   - Verify storage class availability

---

## KubeNodeNotReady

### Description

A Kubernetes node is not in the Ready state. Pods on this node may be affected.

### Severity

**Critical** - Immediate attention required

### Trigger Condition

```promql
kube_node_status_condition{condition="Ready",status="true"} == 0
```

### Investigation Steps

1. **Check node status**

   ```bash
   kubectl get nodes
   kubectl describe node <node-name>
   ```

2. **Check node conditions**

   ```bash
   kubectl get node <node-name> -o jsonpath='{.status.conditions[*]}'
   ```

3. **Check kubelet status (if accessible)**

   ```bash
   ssh <node> systemctl status kubelet
   ssh <node> journalctl -u kubelet --since="10 minutes ago"
   ```

4. **Check for resource pressure**
   ```bash
   kubectl describe node <node-name> | grep -A10 "Conditions:"
   ```

### Remediation Actions

1. **If node is recoverable:**

   ```bash
   # Restart kubelet
   ssh <node> sudo systemctl restart kubelet
   ```

2. **If node is unrecoverable:**

   ```bash
   # Cordon and drain
   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

   # Replace node (EKS/GKE/AKS specific)
   ```

3. **If disk pressure:**
   - Clean up unused images
   - Expand disk or clean logs

4. **If memory pressure:**
   - Evict low-priority pods
   - Add capacity to cluster

### Escalation Path

1. On-call SRE
2. Platform team
3. Cloud provider support (if hardware issue)

---

## KubeNodeUnreachable

### Description

A Kubernetes node cannot be reached by the control plane.

### Severity

**Critical** - Immediate attention required

### Investigation Steps

1. **Check node status**

   ```bash
   kubectl get nodes
   ```

2. **Check cloud provider console**
   - Verify instance is running
   - Check network connectivity
   - Review security groups

3. **Check VPC and networking**
   - Verify VPC routes
   - Check NAT gateway status
   - Review network ACLs

### Remediation Actions

1. **If network issue:**
   - Check security group rules
   - Verify VPC configuration
   - Review network policies

2. **If instance crashed:**
   - Attempt instance restart via cloud console
   - If unrecoverable, terminate and replace

---

## KubeCPUOvercommit

### Description

The cluster has overcommitted CPU - sum of CPU requests exceeds available
capacity.

### Severity

**Warning** - Plan capacity increase

### Trigger Condition

```promql
sum(kube_pod_container_resource_requests{resource="cpu"})
/
sum(kube_node_status_allocatable{resource="cpu"}) > 1
```

### Investigation Steps

1. **Check cluster CPU allocation**

   ```bash
   kubectl describe nodes | grep -A10 "Allocated resources"
   ```

2. **Find highest CPU consumers**
   ```bash
   kubectl top pods --all-namespaces --sort-by=cpu | head -20
   ```

### Remediation Actions

1. **Add cluster capacity:**
   - Scale up node group
   - Add new node pool

2. **Optimize requests:**
   - Review pod CPU requests
   - Right-size based on actual usage

---

## KubeMemoryOvercommit

### Description

The cluster has overcommitted memory - sum of memory requests exceeds available
capacity.

### Severity

**Warning** - Plan capacity increase

### Investigation Steps

1. **Check cluster memory allocation**

   ```bash
   kubectl describe nodes | grep -A10 "Allocated resources"
   ```

2. **Find highest memory consumers**
   ```bash
   kubectl top pods --all-namespaces --sort-by=memory | head -20
   ```

### Remediation Actions

Same as CPU overcommit - add capacity or optimize requests.

---

## NodeDiskRunningFull

### Description

Node disk space is running low (< 15% free).

### Severity

**Critical** - Immediate attention required

### Trigger Condition

```promql
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
```

### Investigation Steps

1. **Check disk usage**

   ```bash
   df -h
   du -sh /* | sort -hr | head -20
   ```

2. **Check for large log files**

   ```bash
   find /var/log -type f -size +100M
   ```

3. **Check container images**
   ```bash
   docker system df
   crictl images | wc -l
   ```

### Remediation Actions

1. **Clean up Docker/containerd images**

   ```bash
   docker system prune -a
   crictl rmi --prune
   ```

2. **Clean up old logs**

   ```bash
   journalctl --vacuum-time=7d
   find /var/log -type f -name "*.gz" -mtime +7 -delete
   ```

3. **Expand disk (cloud-specific)**
   - EKS: Modify EBS volume, then resize filesystem
   - GKE: Edit node pool disk size
   - AKS: Scale VM disk

---

## NodeHighCPUUsage

### Description

Node CPU utilization is consistently high (> 90%).

### Severity

**Warning** - Investigate and plan capacity

### Trigger Condition

```promql
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
```

### Investigation Steps

1. **Identify high CPU processes**

   ```bash
   kubectl top pods -A --sort-by=cpu | head -20
   ```

2. **Check for CPU throttling**
   ```promql
   rate(container_cpu_cfs_throttled_seconds_total[5m])
   ```

### Remediation Actions

1. **Scale out workloads** to distribute load
2. **Increase node size** if consistently overloaded
3. **Optimize application** if specific service is the cause

---

## NodeHighMemoryUsage

### Description

Node memory utilization is high (> 90%).

### Severity

**Warning** - Risk of OOM conditions

### Trigger Condition

```promql
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
```

### Investigation Steps

1. **Identify high memory pods**

   ```bash
   kubectl top pods -A --sort-by=memory | head -20
   ```

2. **Check for memory leaks**
   ```promql
   container_memory_usage_bytes{namespace=~"lilith.*"}[1h]
   ```

### Remediation Actions

1. **Scale or restart memory-leaking pods**
2. **Increase node size**
3. **Add memory limits** to prevent runaway processes

---

## General Commands

```bash
# Cluster overview
kubectl get nodes -o wide
kubectl top nodes

# Pod status across namespaces
kubectl get pods -A | grep -v Running

# Recent events
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

# Resource allocation
kubectl describe nodes | grep -A20 "Allocated resources"
```

## Contact Information

| Role          | Contact                       |
| ------------- | ----------------------------- |
| On-Call SRE   | #sre-oncall (Slack)           |
| Platform Team | #platform-team (Slack)        |
| Cloud Team    | #cloud-infrastructure (Slack) |

## References

- [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug/)
- [Node Health Monitoring](../observability/node-health.md)
- [Capacity Planning](../infrastructure/capacity-planning.md)
