Back to all scenarios
Scenario #187
Networking
K8s v1.20, On-Premise
Flapping Node Network Connectivity Due to MTU Mismatch
Nodes in the cluster were flapping due to mismatched MTU settings between Kubernetes and the underlying physical network, causing intermittent network connectivity issues.
Find this helpful?
What Happened
The physical network’s MTU was configured differently from the MTU settings in the Kubernetes CNI plugin, causing packet fragmentation. As a result, node-to-node communication was sporadic.
Diagnosis Steps
- 1Used kubectl describe node and checked the node’s network configuration.
- 2Verified the MTU settings in the physical network and compared them to the Kubernetes settings, which were mismatched.
Root Cause
The mismatch in MTU settings caused fragmentation, resulting in unreliable connectivity between nodes.
Fix/Workaround
• Updated the Kubernetes network plugin's MTU setting to match the physical network MTU.
• Restarted the affected nodes and validated the network stability.Lessons Learned
Ensure that the MTU setting in the CNI plugin matches the physical network's MTU to avoid connectivity issues.
How to Avoid
- 1Always verify the MTU settings in both the physical network and the CNI plugin during cluster setup.
- 2Include network performance testing in your cluster validation procedures.