Back to all scenarios
Scenario #20
Cluster Management
K8s v1.24, DaemonSet-based node config

Node Labels Accidentally Overwritten by DaemonSet

A DaemonSet used for node labeling overwrote existing labels used by schedulers.

Find this helpful?
What Happened

A platform team deployed a DaemonSet that set node labels like zone=us-east, but it overwrote custom labels like gpu=true.

Diagnosis Steps
  • 1Pods no longer scheduled to GPU nodes.
  • 2kubectl get nodes --show-labels showed gpu label missing.
  • 3Checked DaemonSet script – labels were overwritten, not merged.
Root Cause

Label management script used kubectl label node <node> key=value --overwrite, removing other labels.

Fix/Workaround
• Restored original labels from backup.
• Updated script to merge labels.
Lessons Learned

Node labels are critical for scheduling decisions.

How to Avoid
  • 1Use label merging logic (e.g., fetch current labels, then patch).
  • 2Protect key node labels via admission controllers.
Previous ScenarioNext Scenario