Back to all scenarios
Scenario #37
Cluster Management
K8s v1.25, self-hosted
Outdated Admission Webhook Blocking All Deployments
A stale mutating webhook caused all deployments to fail due to TLS certificate errors.
Find this helpful?
What Happened
The admission webhook had expired TLS certs, causing validation errors on all resource creation attempts.
Diagnosis Steps
- 1Created a dummy pod and observed webhook errors.
- 2Checked logs of the webhook pod – found TLS handshake failures.
Root Cause
Webhook server was down due to expired TLS cert.
Fix/Workaround
• Renewed cert and redeployed webhook.
• Disabled webhook temporarily for emergency deployments.Lessons Learned
Webhooks are gatekeepers – they must be monitored.
How to Avoid
- 1Rotate webhook certs using cert-manager.
- 2Alert on webhook downtime or errors.