Back to all scenarios
Scenario #335
Storage
Kubernetes v1.21, Longhorn
Zombie VolumeAttachment Blocks New PVC
After a node crash, a VolumeAttachment object was not garbage collected, blocking new PVCs from attaching.
Find this helpful?
What Happened
Application tried to use the volume, but Longhorn saw the old attachment from a dead node and refused reattachment.
Diagnosis Steps
- 1Listed VolumeAttachment resources — found one pointing to a non-existent node.
- 2Longhorn logs: volume already attached to another node.
- 3Node was removed forcefully.
Root Cause
VolumeAttachment controller did not clean up orphaned entries on node deletion.
Fix/Workaround
• Manually deleted VolumeAttachment.
• Restarted CSI pods to refresh state.Lessons Learned
Controller garbage collection is fragile post-node failure.
How to Avoid
- 1Use node lifecycle hooks to detach volumes gracefully.
- 2Alert on dangling VolumeAttachments.