In the world of software development, there are bugs, and then there are legends. This is the story of a legend. A ghost bug that haunted engineers at AWS for three years, until it was finally caught.
Imagine an internal service. Its state is a file that stores data in the form of an associative array. Now, imagine that from this file, a critical entry mysteriously disappears without a trace. Every time this happened, part of the system was paralyzed. Data processing ground to a halt until an on-call engineer manually found the missing entry and restored it.
The First Hunters
The team, of course, didn't sit idly by. They launched full-scale investigations:
1. Code Review: The code was meticulously combed, but to no avail. The bug left no trace.
2. Traps and Snares: A protective mechanism was added. Immediately after saving the file, a special code would check if all the data was present. If not, it would attempt to write the data again.
3. Monitoring: Metrics and alarms were set up to detect the problem as early as possible.
These measures helped. They managed to wound the beast, but not kill it. The frequency of its appearances decreased to once every six months. But every six months, the ghost returned, and another team of investigators came up empty-handed.
New Blood
And then, I joined the project. Time passed, and this "ghost" knocked on my door as well - I was assigned to investigate the latest incident. Like my predecessors, I delved into the code. It was quite an adventure: the code was convoluted, non-linear, with a ton of dependencies - the perfect breeding ground for a tricky bug.
After many hours poring over logs, I stumbled upon something strange. In that very protective mechanism, which was supposed to check data integrity, there was its own bug! A sneaky bug that only triggered very rarely (only during the first save of the file after the host started) and prevented the check from working properly. I fixed it, feeling like a winner. But the joy was short-lived. I had caught not the beast, but only its shadow. I fixed the patch, but not the root cause!
The Epiphany Over Coffee
They say that our brains can solve problems in the background. You can go about your business, and it continues to process options. And then, one morning, almost like in a movie, it hit me. I suddenly realized where the race condition might be hiding in the code. Two threads were trying to work with the same data array simultaneously, and one of them was sometimes losing, taking that very entry with it into oblivion.
The rest was a matter of technique. I wrote a tiny unit test that reproduced the bug with 100% certainty. It was like seeing the ghost in the camera lens. And then - one elegant move: replacing the vulnerable HashMap with a thread-safe ImmutableMap.
The ghost was caught. The bug never returned again.
The Moral of This Detective Story:
1.
HashMap and multithreading are a dangerous mix. If you are writing and reading from a HashMap in multiple threads simultaneously, be prepared for the possibility that data may disappear or become null. It's not "if," it's "when."2. Workarounds are a double-edged sword. They can temporarily save the situation, but they make finding the root cause much more difficult. Sometimes it's simpler and cheaper to rewrite the code from scratch than to untangle a mess of hacks.
3. Workarounds can have their own bugs. By adding one layer of protection, you may inadvertently create a new vulnerability.
4. Race conditions are the sneakiest enemy. They are easy to introduce by accident and incredibly difficult to catch because they manifest rarely and unpredictably.
So, the next time you encounter an "unexplained" bug, remember: you might just be hunting a ghost.
#ru #en #bugs