Back to all scenarios
Scenario #488
Scaling & Load
Kubernetes v1.23, AWS EKS
Failed Scaling due to Insufficient Node Capacity for StatefulSets
Scaling failed because the node pool did not have sufficient capacity to accommodate new StatefulSets.
Find this helpful?
What Happened
When trying to scale a StatefulSet, the system couldn't allocate enough resources on the available nodes, causing scaling to fail.
Diagnosis Steps
- 1Checked resource availability across nodes and found that there wasn’t enough storage or CPU capacity for StatefulSet pods.
- 2Observed that the cluster's persistent volume claims (PVCs) were causing resource constraints.
Root Cause
Inadequate resource allocation, particularly for persistent volumes, when scaling StatefulSets.
Fix/Workaround
• Increased the node pool size and resource limits for the StatefulSets.
• Rescheduled PVCs and balanced the resource requests more effectively across nodes.Lessons Learned
StatefulSets require careful resource planning, especially for persistent storage.
How to Avoid
- 1Regularly monitor resource utilization, including storage, during scaling events.
- 2Ensure that node pools have enough capacity for StatefulSets and their associated storage requirements.