Recently, I encountered a situation related to the qualification of a new type of server for our service. Overall, everything was working perfectly, but occasionally, a very strange error appeared in the logs: multiplying two positive constants resulted in a negative value, which led to a logic error and an exception. Imagine multiplying
128 by 1.5 and instead of 192, getting -2147483647.This problem only manifested on servers of this specific type, and it was not deterministic. Sometimes, there would be only one error in a day, while other times, there would be no problems for a whole week.
To solve the problem, it was necessary to create a deterministic reproduction scenario. Within the test environment, I took the following steps:
1. I updated the application code to make the code that caused the error run a thousand times more frequently. As a result, the error started occurring every 10 minutes.
2. I gradually removed sections of the application, while simultaneously checking if the reproducibility of the error was maintained.
3. I repeated step 2 until I reached the minimal code that caused the error.
In the final version, the code to reproduce the error consisted of the following sequence of steps:
1. Calculating a CRC32 checksum.
2. Attempting to multiply two constants.
3. Comparing the result of the multiplication with the expected value.
Incredibly, the multiplication of the two constants consistently produced random results. With the motto "that's all we can do here", I documented the error and passed it on to the JDK team.
Many well-known Java experts, such as Alexey Shipilov, Andrey Panging, and others, quickly joined the analysis of this bug. It turned out that the root of the problem was as follows:
1. For the new type of processor, a intrinsic function for CRC32/CRC32C was implemented, which significantly improved the performance of the algorithm.
2. It was discovered that this intrinsic function used
kernel_crc32_common_fold_using_crypto_pmull, which, in turn, could write random data to registers V8-V15 of the processor if they were not previously saved.3. According to the specification, these registers should have been saved before the call and restored after. However, this requirement was not implemented.
As a result, the JDK team promptly filed a bug in OpenJDK and provided a fix.
The moral of the story:
1. Even old, well-functioning code can suddenly encounter unpredictable problems when migrating to a new type of processor.
2. Identifying the source of the problem and creating a deterministic reproduction scenario are crucial for successfully resolving technical issues.
3. Thorough testing is necessary, especially when working with complex elements like intrinsics.
4. Cases like this highlight that the JDK team is capable of quickly and effectively resolving such bugs, which underscores the importance of active collaboration within the expert community to ensure the stability and reliability of software.
How about that, Elon Musk?
#work #programming #bugs