Back to all scenarios
Scenario #171
Networking
K8s v1.22, AWS EKS
Network Bottleneck Due to Overutilized Network Interface
Network bottleneck occurred due to overutilization of a single network interface on the worker nodes.
Find this helpful?
What Happened
The worker nodes were using a single network interface to handle both pod traffic and node communication. The high volume of pod traffic caused the network interface to become overutilized, resulting in slow communication.
Diagnosis Steps
- 1Checked the network interface metrics using AWS CloudWatch and found that the interface was nearing its throughput limit.
- 2Used kubectl top node and observed high network usage on the affected nodes.
Root Cause
The network interface on the worker nodes was not properly partitioned to handle separate types of traffic, leading to resource contention.
Fix/Workaround
• Added a second network interface to the worker nodes for pod traffic and node-to-node communication.
• Reconfigured the nodes to distribute traffic across the two interfaces.Lessons Learned
Proper network interface design is crucial for handling high traffic loads and preventing bottlenecks.
How to Avoid
- 1Design network topologies that segregate different types of traffic (e.g., pod traffic, node communication).
- 2Regularly monitor network utilization and scale resources as needed.