Back to all scenarios
Scenario #15
Cluster Management
K8s v1.21, production cluster with HPA and PDB
Node Drain Fails Due to PodDisruptionBudget Deadlock
kubectl drain never completed because PDBs blocked eviction.
Find this helpful?
What Happened
A deployment had minAvailable: 2 in PDB, but only 2 pods were running. Node drain couldn’t evict either pod without violating PDB.
Diagnosis Steps
- 1Ran kubectl describe pdb <name> – saw AllowedDisruptions: 0.
- 2Checked deployment and replica count.
- 3Tried drain – stuck on pod eviction for 10+ minutes.
Root Cause
PDB guarantees clashed with under-scaled deployment.
Fix/Workaround
• Temporarily edited PDB to reduce minAvailable.
• Scaled up replicas before drain.Lessons Learned
PDBs require careful coordination with replica count.
How to Avoid
- 1Validate PDBs during deployment scale-downs.
- 2Create alerts for PDB blocking evictions.