Bug #1: AKS, Metrics Server, and the Elusive Timeout on HDD (or How I Lost 3 Days of My Life)
My first story in the "you can't catch us, but we'll drive you crazy" series takes us back to 2020. At the time, I was working on a project where we were automating the deployment of environments, and one of the tasks was to install Azure Kubernetes Service (AKS). And that's where the trouble began.
It seems simple enough: a fresh installation of AKS using Terraform? But no, this thing would occasionally timeout. And this would lead to a complete halt of the entire environment configuration. Imagine: automation, everything's going smoothly, and then BAM! – you have to wait for something to come back online somewhere, or restart the entire process. Precious time was lost, and with it, my patience.
Investigation
After many hours (or rather, days!) of digging, it became clear that the root of the problem lay in the Metrics Server. This important component would freeze immediately after the installation of Ingress. The problem was so obscure that I even gave up and went for help on Stack Overflow. Honestly, I spent no less than three working days on this. That's an eternity when you're trying to figure out why your Metrics Server is sabotaging you.
Anatomy of the Bug: HDD, I/O, and the "Secret Life" of the Load Balancer
And here's the climax! In the end, after a million guesses and tests, the cause was found. It turned out that AKS automatically installs an Ingress Load Balancer after the installation of Ingress. This in itself isn't a problem, but here's the catch: the installation of this Load Balancer starts a few minutes later and, attention, begins to actively affect the Kubernetes API Server.
And since we were using HDD storage (yes, it was a test environment, but it was still supposed to work!), the API server would freeze due to wild spikes in Disk I/O. This, in turn, led to errors in the operation of
tunnelfront (oh, that tunnelfront!). And the Metrics Server, which, as you remember, actively communicates with the Kubernetes API, would receive timeouts and start crashing in a loop. I even managed to find a corresponding bug on GitHub, so I wasn't alone in my suffering.Solution: Unexpected Liveness Tests
So, how did we overcome this? The solution turned out to be surprisingly simple and unexpected. We solved the problem by automatically patching the Metrics Server Deployment resource with `liveness/readiness probes`. You won't believe it, but they weren't there by default!
The presence of these simple checks led to the timely restart of the Metrics Server itself if it started to freeze, and got it out of this vicious cycle of timeouts. And there it was, the Metrics Server, alive and well again!
Have you ever experienced similar "survival races" with bugs? Share your stories! What do you think about this situation?
#ru #en #bugs