Back to all scenarios
Scenario #123
Networking
K8s v1.21, Azure AKS
Pod Network Performance Degradation After Node Upgrade
Network performance degraded after an automatic node upgrade, causing latency in pod communication.
Find this helpful?
What Happened
After an upgrade to a node pool, there was significant latency in network communication between pods, impacting application performance.
Diagnosis Steps
- 1Checked pod network latency using ping and found increased latency between pods.
- 2Examined node and CNI logs, identifying an issue with the upgraded network interface drivers.
Root Cause
Incompatible network interface drivers following the node upgrade.
Fix/Workaround
• Rolled back the node upgrade and manually updated the network interface drivers on the nodes.
• Verified that network performance improved after driver updates.Lessons Learned
Be cautious when performing automatic upgrades in production environments.
How to Avoid
- 1Manually test upgrades in a staging environment before applying them to production.
- 2Ensure compatibility of network drivers with the Kubernetes version being used.