Back to all scenarios
Scenario #331
Storage
Kubernetes v1.26, MinIO CSI driver, Ceph RGW backend
ReadAfterWrite Inconsistency on Object Store-Backed CSI
Applications experienced stale reads immediately after writing to the same file via CSI mount backed by an S3-like object store.
Find this helpful?
What Happened
A distributed app wrote metadata and then read it back to validate—however, the file content was outdated due to eventual consistency in object backend.
Diagnosis Steps
- 1Logged file hashes before and after write — mismatch seen.
- 2Found underlying storage was S3-compatible with eventual consistency.
- 3CSI driver buffered writes asynchronously.
Root Cause
Object store semantics (eventual consistency) not suitable for synchronous read-after-write patterns.
Fix/Workaround
• Introduced write barriers and retry logic in app.
• Switched to CephFS for strong consistency.Lessons Learned
Object store-backed volumes need strong consistency guards.
How to Avoid
- 1Avoid using S3-style backends for workloads expecting POSIX semantics.
- 2Use CephFS, NFS, or block storage for transactional I/O.