Back to all scenarios
Scenario #301
Storage
Kubernetes v1.22, EBS CSI Driver on EKS
PVC Stuck in Terminating State After Node Crash
A node crash caused a PersistentVolumeClaim (PVC) to be stuck in Terminating, blocking pod deletion.
Find this helpful?
What Happened
The node hosting the pod with the PVC crashed and never returned. The volume was still attached, and Kubernetes couldn’t cleanly unmount or delete it.
Diagnosis Steps
- 1Described the PVC: status was Terminating.
- 2Checked finalizers on the PVC object.
- 3Verified the volume was still attached to the crashed node via AWS Console.
Root Cause
The volume attachment record wasn’t cleaned up due to the ungraceful node failure.
Fix/Workaround
• Manually removed the PVC finalizers.
• Used aws ec2 detach-volume to forcibly detach.Lessons Learned
Finalizers can block PVC deletion in edge cases.
How to Avoid
- 1Use the external-attacher CSI sidecar with leader election.
- 2Implement automation to detect and clean up stuck attachments.