Load Balancer Health Checks Failing After Pod Update

Load balancer health checks failed after updating a pod due to incorrect readiness probe configuration.

Find this helpful?

What Happened

After deploying a new version of the application, the load balancer’s health checks started failing, causing traffic to be routed to unhealthy pods.

Diagnosis Steps

1Reviewed the load balancer logs and observed failed health checks on newly deployed pods.
2Inspected the pod’s readiness probe and found that it was configured incorrectly, leading to premature success.

Root Cause

Incorrect readiness probe causing the pod to be marked healthy before it was ready to serve traffic.

Fix/Workaround

• Corrected the readiness probe configuration to reflect the actual application startup time.
• Redeployed the updated pods and verified that they passed the health checks.

Lessons Learned

Always validate readiness probes after updates to avoid traffic disruption.

How to Avoid

1Test readiness probes extensively during staging before updating production.
2Implement rolling updates to avoid downtime during pod updates.

Previous Scenario Next Scenario