Back to all scenarios
Scenario #65
Cluster Management
K8s v1.22, AWS EKS
Cluster Performance Degradation Due to Excessive Logs
Cluster performance degraded because of excessive logs being generated by applications, leading to high disk usage.
Find this helpful?
What Happened
Excessive log output from applications filled up the disk, slowing down the entire cluster.
Diagnosis Steps
- 1Monitored disk usage and found that logs were consuming most of the disk space.
- 2Identified the affected applications by reviewing the logging configuration.
Root Cause
Applications were logging excessively, and log rotation was not properly configured.
Fix/Workaround
• Configured log rotation for the affected applications.
• Reduced the verbosity of the logs in application settings.
Lessons Learned
Proper log management and rotation are essential to avoid filling up disk space and impacting cluster performance.
How to Avoid
- 1Configure log rotation and retention policies for all applications.
- 2Monitor disk usage and set up alerts for high usage.