Back to all scenarios
Scenario #364
Storage
Kubernetes v1.25, custom CSI driver
CSI Driver Crashes When Node Resource Is Deleted Prematurely
Deleting a node object before the CSI driver detached volumes caused crash loops.
Find this helpful?
What Happened
Admin manually deleted a node before volume detach completed.
Diagnosis Steps
- 1CSI logs showed panic due to missing node metadata.
- 2Pods remained in Terminating.
Root Cause
Driver attempted to clean up mounts from a non-existent node resource.
Fix/Workaround
• Waited for CSI driver to timeout and self-recover.
• Rebooted node to forcibly detach volumes.Lessons Learned
Node deletion should follow strict lifecycle policies.
How to Avoid
- 1Use node cordon + drain before deletion.
- 2Monitor CSI cleanup completion before proceeding.