Back to all scenarios
Scenario #84
Cluster Management
K8s v1.21, AWS EKS
Pod Eviction Due to Node Disk Pressure
Pods were evicted due to disk pressure on the node, causing service interruptions.
Find this helpful?
What Happened
A node ran out of disk space due to logs and other data consuming the disk, resulting in pod evictions.
Diagnosis Steps
- 1Checked node resource usage and found disk space was exhausted.
- 2Reviewed pod eviction events and found they were due to disk pressure.
Root Cause
The node disk was full, causing the kubelet to evict pods to free up resources.
Fix/Workaround
• Increased disk capacity on the affected node.
• Cleared unnecessary logs and old data from the disk.Lessons Learned
Ensure adequate disk space is available, especially for logging and temporary data.
How to Avoid
- 1Monitor disk usage closely and set up alerts for disk pressure.
- 2Implement log rotation and clean-up policies to avoid disk exhaustion.