Back to all scenarios
Scenario #464
Scaling & Load
Kubernetes v1.25, IBM Cloud
Latency Spikes During Horizontal Pod Scaling
Latency spikes occurred during horizontal pod scaling due to inefficient pod distribution.
Find this helpful?
What Happened
Horizontal pod scaling caused latency spikes as the traffic was unevenly distributed between pods, some of which were underutilized while others were overloaded.
Diagnosis Steps
- 1Reviewed traffic distribution and pod scheduling, which revealed that the load balancer did not immediately update routing configurations.
- 2Found that newly scaled pods were not receiving traffic promptly.
Root Cause
Delayed update in load balancer routing configuration after scaling.
Fix/Workaround
• Configured load balancer to refresh routing rules as soon as new pods were scaled up.
• Implemented readiness probes to ensure that only fully initialized pods were exposed to traffic.Lessons Learned
Load balancer reconfiguration must be synchronized with pod scaling events.
How to Avoid
- 1Use automatic load balancer updates during scaling events.
- 2Configure readiness probes to ensure proper pod initialization before they handle traffic.