Table of Contents

runbooks:coustom_alerts:KubernetesNodeOutOfPodCapacity

KubernetesNodeOutOfPodCapacity

Meaning

This alert is triggered when a Kubernetes node reaches more than 90% of its pod capacity for more than 2 minutes. It indicates that the node has almost no free allocatable pod slots left.

Impact

A node running out of pod capacity can cause:

This alert is marked warning, as it may precede node-level failures or application disruptions.

Diagnosis

Check node pod allocation:

kubectl get nodes -o wide
kubectl describe node <NODE_NAME>

Check running pods on the node:

kubectl get pods -o wide --all-namespaces | grep <NODE_NAME>

Check node allocatable pods:

kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable.pods}'

Check cluster-wide pod distribution:

kubectl get pods --all-namespaces -o wide

Possible Causes

Mitigation

  1. Review and redistribute workloads across nodes
  2. Scale out the cluster by adding more nodes
  3. Remove unnecessary pods or workloads from the node
  4. Adjust DaemonSets or affinity/anti-affinity rules
  5. Enable or tune Cluster Autoscaler if available

Escalation