Back to all scenarios
Scenario #350
Storage
Kubernetes v1.26, CSI, NVMes
Node Crash Triggers Volume Remount Loop
After a node crash, volume remount loop occurred due to conflicting device paths.
Find this helpful?
What Happened
Volume had a static device path cached in CSI driver. Upon node recovery, OS assigned a new device path. CSI couldn't reconcile.
Diagnosis Steps
- 1CSI logs: device path not found.
- 2Pod remained in ContainerCreating.
- 3OS showed volume present under different path.
Root Cause
CSI assumed static device path, OS changed it post-reboot.
Fix/Workaround
• Added udev rules for consistent device naming.
• Restarted CSI daemon to detect new device path.Lessons Learned
Relying on device paths can break persistence.
How to Avoid
- 1Use device UUIDs or filesystem labels where supported.
- 2Restart CSI pods post-reboot events.