Back to all scenarios
Scenario #68
Cluster Management
K8s v1.22, AWS EKS
Cluster Inaccessibility Due to API Server Throttling
Cluster became inaccessible due to excessive API server throttling caused by too many concurrent requests.
Find this helpful?
What Happened
The API server started throttling requests because the number of concurrent API calls exceeded the available limit.
Diagnosis Steps
- 1Monitored API server metrics and identified a high rate of incoming requests.
- 2Checked client application logs and observed excessive API calls.
Root Cause
Clients were making too many API requests in a short period, exceeding the rate limits of the API server.
Fix/Workaround
• Throttled client requests to reduce API server load.
• Implemented exponential backoff for retries in client applications.Lessons Learned
Avoid overwhelming the API server with excessive requests and implement rate-limiting mechanisms.
How to Avoid
- 1Implement API request throttling and retries in client applications.
- 2Use rate-limiting tools like kubectl to monitor API usage.