Okay, so bug #1. I encountered this in 2020 while working on a peculiar issue: sometimes, when setting up a fresh Azure Kubernetes Service cluster, the installation would occasionally time out. Since the installation was automated using Terraform, this timeout would halt the configuration of the entire environment.

After investigating, it became clear that the Metrics Server would freeze immediately after the Ingress was installed. The problem was so obscure that I even asked a question on Stack Overflow. As far as I remember, I spent at least 3 working days trying to figure this out.

Eventually, it was discovered that AKS automatically installs an Ingress Load Balancer in the cluster after the Ingress is installed. The installation of the Load Balancer starts a few minutes later and affects the Kubernetes API Server. Because HDD storage was being used (this was a test environment), the API server would hang due to spikes in Disk I/O, causing errors in the operation of tunnelfront. Since the Metrics Server uses the Kubernetes API, it would crash in a loop with timeouts. I even managed to find a corresponding bug report on GitHub.

The problem was resolved by automatically patching the Metrics Server Deployment resource by adding liveness/readiness probes to it (it was quite surprising that they weren't included by default). The presence of these probes caused the Metrics Server to restart, resolving the hang.

What do you think of this bug? What are your thoughts?
I hope I haven't overwhelmed you with too many technical details 😄

#work #programming #bugs