Back to all scenarios
Scenario #165
Networking
K8s v1.21, AWS EKS
Pod Network Latency Due to Overloaded Kubernetes Network Interface
Pod network latency increased due to an overloaded network interface on the Kubernetes nodes.
Find this helpful?
What Happened
A sudden increase in traffic caused the network interface on the nodes to become overloaded, leading to high network latency between pods and degraded application performance.
Diagnosis Steps
- 1Used kubectl top node to observe network interface metrics and saw high network throughput and packet drops.
- 2Checked AWS CloudWatch metrics and confirmed that the network interface was approaching its maximum throughput.
Root Cause
The network interface on the nodes was unable to handle the high network traffic due to insufficient capacity.
Fix/Workaround
• Increased the network bandwidth for the AWS EC2 instances hosting the Kubernetes nodes.
• Used network policies to limit traffic to critical pods and avoid overwhelming the network interface.Lessons Learned
Ensure that Kubernetes nodes are provisioned with adequate network capacity for expected traffic loads.
How to Avoid
- 1Monitor network traffic and resource utilization at the node level.
- 2Scale nodes appropriately or use higher-bandwidth instances for high-traffic workloads.