Bug #4. The history of this bug began long before I joined AWS and continued for three years.

One of the internal services had a problem: for some reason, one record was disappearing from a file that stored a serialized associative array. This data loss caused partial data processing to be blocked until an operator manually discovered the missing record and added it back to the file.

The problem was serious, so the service team tried to fix it in every possible way:
1. Searching for the bug in the code was unsuccessful.
2. They added additional checks after saving the file to ensure that all data was saved. If data was missing, a retry was attempted.
3. They added some metrics and configured alarms based on them.

The frequency of the bug decreased to once every six months. So, every six months, there was an attempt to understand the bug, and each time it failed.

When I joined the project, I also encountered this problem. Like others, I was assigned to investigate the root cause of the latest occurrence. In turn, I also tried to find the bug in the code. This was not easy: the code was not linear, was not written in the best way, and had many dependencies.

After a long analysis of the logs, I discovered that there was a bug in the code that was added as an additional check for saved data. This bug occurred quite rarely (only during the first save of the file after the host started) and caused the data not to be checked correctly. The bug was fixed, and the problem should no longer occur. But this was only a bug in the workaround, not the root cause.

They say that the brain can work on a problem in the background. For example, when you wake up and suddenly have a bright idea that you hadn't thought of before. The same thing happened to me. Suddenly, I realized a place in the code where there might be a race condition when writing data to that associative array. As a result, the bug was reproduced using a unit test, fixed by using an ImmutableMap, and never occurred again.

The moral of the story:
1. If you are writing to and reading from a HashMap simultaneously from multiple threads, be prepared for data to disappear or for you to receive null instead of a value.
2. If you add a workaround for a bug in the code, be prepared for it to be so difficult to reproduce and detect that it will be easier to rewrite the code from scratch.
3. There can also be bugs in workarounds for bugs.
4. Race conditions are easy to introduce but very difficult to find.

#work #programming #bugs