HostOutOfDiskSpace

This alert is triggered when a host node’s disk has less than 10% free space on any filesystem (excluding tmpfs, fuse, cifs, nfs) for more than 2 minutes. It indicates that the host is running low on disk space, which may cause system or application failures.

Low disk space can cause:

• Pod evictions due to inability to write logs or data
• Application failures
• Node instability or crashes
• Kubernetes scheduling failures for pods with persistent volume requirements
• Increased latency or I/O errors

This alert is critical, as disk space exhaustion can immediately impact production workloads.

Check disk usage:

df -h
df -i
lsblk

Check disk space per mountpoint:

du -sh /var/lib/kubelet/*
du -sh /home/*

Check pods consuming disk:

kubectl get pvc --all-namespaces
kubectl describe pod <POD_NAME> -n <NAMESPACE>

Check node events:

kubectl get events --sort-by=.lastTimestamp

• Large log files or temporary files
• Full persistent volumes
• Backup jobs or batch jobs filling disks
• Container images not cleaned up
• Disk size too small for workload

1. Clean up unused files, logs, or images
2. Rotate and compress logs
3. Move non-critical data to another storage
4. Evict or reschedule non-critical pods
5. Increase disk capacity if possible

Drain node if needed:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data
kubectl uncordon <NODE_NAME>

• Escalate if disk space remains below 10% for extended periods
• Page on-call engineer if production services are impacted
• Treat multiple nodes with low disk space as a cluster-level incident

• HighDiskUsage
• HighDiskIOWait
• KubernetesNodeDiskPressure
• HostUnusualDiskReadRate

• Grafana → Node Disk Usage
• Grafana → Node Exporter Disk Metrics