Back to all scenarios
Scenario #31
Cluster Management
K8s v1.22, on-prem, kubeadm
Node Clock Drift Causing Authentication Failures
Authentication tokens failed across the cluster due to node clock skew.
Find this helpful?
What Happened
Token-based authentication failed for all workloads and kubectl access due to time drift between worker nodes and the API server.
Diagnosis Steps
- 1Ran kubectl logs and found expired token errors.
- 2Checked node time using date on each node – found significant drift.
- 3Verified NTP daemon status – not running.
Root Cause
NTP daemon disabled on worker nodes.
Fix/Workaround
• Re-enabled and restarted NTP on all nodes.
• Synchronized system clocks manually.Lessons Learned
Time synchronization is critical for certificate and token-based auth.
How to Avoid
- 1Ensure NTP or chrony is enabled via bootstrap configuration.
- 2Monitor time drift via node-exporter.