Back to all scenarios
Scenario #325
Storage
Kubernetes v1.23, NFS PVC shared by 2 pods
Data Corruption on Shared Volume With Two Pods
Two pods writing to the same volume caused inconsistent files and data loss.
Find this helpful?
What Happened
Both pods ran jobs writing to the same output files. Without file locking, one pod overwrote data from the other.
Diagnosis Steps
- 1Logs showed incomplete file writes.
- 2File hashes changed mid-run.
- 3No mutual exclusion mechanism implemented.
Root Cause
Shared volume used without locking or coordination between pods.
Fix/Workaround
• Refactored app logic to coordinate file writes via leader election.
• Used a queue-based processing system.Lessons Learned
Shared volume access must be controlled explicitly.
How to Avoid
- 1Never assume coordination when using shared volumes.
- 2Use per-pod PVCs or job-level locking.