Today, I want to share a story that still seems one of the craziest in my career. It's about how fundamental laws of mathematics suddenly stopped working on a new type of server in AWS.
Imagine: you're confident in your code, like you are in the laws of physics. But what if one day your server decides that
128 * 1.5 is not 192, but `-2147483647`? Sounds like nonsense, right? But that's exactly what I had to deal with.A Ghost in the Machine
It all started during the qualification of a new type of server for our service. Overall, everything was working fine, but occasionally, errors would pop up in the logs that made my hair stand on end. The application was crashing with an exception because the result of multiplying two positive constants was suddenly negative.
The worst part was that it was a ghost bug. It only appeared on the new servers and was completely unpredictable: sometimes once a day, sometimes not at all for a whole week. How do you debug something that's impossible to catch?
The Bug Hunt
It became clear: to defeat the enemy, you first have to corner it. I had to turn into a detective and force the bug to appear on command.
1. Increase the frequency: First, I modified the code to make the problematic section run a thousand times more often. It worked! The error started appearing consistently every 10 minutes. That's something!
2. Surgical method: Next, I started systematically, piece by piece, removing code from the application, each time checking: "Is the bug still here?"
3. Minimal suspect: I repeated this process until I was left with a tiny, but working, example that consistently broke math.
And then, after many hours of "surgery," I had it. The minimal code that caused the error. It looked like a scene from an absurdist play:
1. Calculate the
CRC32 checksum.2. Multiply two constants.
3. Compare the result with the expected value... and it doesn't match!
Incredibly, but true: after calculating
CRC32, the simplest multiplication started producing random garbage.At that moment, I realized: the problem was deeper than just a bug in the application. With the motto "this is where our powers end!", I carefully documented the finding and passed it on to the real gurus - the JDK team.
The Resolution: What Was It?
Titans of the Java world, such as Alexey Shipilov and Andrey Pangin, joined the investigation. The truth turned out to be both simple and ingenious.
1. Optimization: A special intrinsic was added to Java for new processors to calculate
CRC32. This is a built-in, super-optimized instruction that works much faster than regular code.2. Side effect: It turned out that this fast instruction (
kernel_crc32_common_fold_using_crypto_pmull) had an unpleasant side effect. It used certain processor registers (V8-V15) for its calculations, but "forgot" to save their original state and restore everything as it was.3. Contract violation: According to the ARM processor specifications, these registers should be saved before the call and restored after. This requirement was not met.
As a result, after calling
CRC32, garbage remained in the registers. And the next piece of code (our poor multiplication) picked up this "dirty" data instead of the correct values. Voilà! Get your negative product.The OpenJDK team reacted quickly: a bug JDK-8324874 was created and a fix PR#17701 was released promptly.
The Moral of the Story
This story is a great reminder of several things:
* Nothing is eternal: Even the most stable code can break on new "hardware" in the most unexpected ways.
* Reproducibility is key to success: Finding and localizing a bug is 90% of the work. Creating a deterministic scenario is your main victory in debugging.
* Trust, but verify: Especially when it comes to low-level optimizations like intrinsics.
* The power of community: This story perfectly shows how quickly and effectively the JDK team and the community of experts solve even the most complex problems.
Well, what do you think, Elon Musk?
#ru #en #bugs