Bug #5. It's a bug, but not really.

About a year ago, we encountered an issue in the service I work on. While processing data from another service, we detected data corruption during checksum verification. Repeated attempts to process the data resulted in the same error. This is not something you typically expect when working on a standard application.

After manually analyzing the data, we discovered a bit flip – for some reason, the value of one of the bits had been inverted. We corrected the data, the checksum matched, and processing continued.

But what caused the bit flip? We traced the data path through all the services, from its origin. This is a fairly complex task, especially when you have hundreds of services and tens of thousands of servers. Eventually, we identified a network card that was occasionally transmitting incorrect data (a hardware problem). The device was replaced with a new one, and the problem hasn't recurred since.

But why didn't any of the previous services detect the problem? It turned out that not all services validate all checksums during processing. In this case, the checksum was "nested," meaning it was a checksum for a portion of the data within other data.

To increase reliability, we decided to implement checksum bracketing – an approach where the checksum for the data is calculated and verified throughout the entire data processing pipeline. Here's how it might look:
1. The service receives data and a checksum from another service: D + C.
2. The service validates the checksum and ensures that the data was received correctly: checksum(D) == C.
3. The service transforms the data and calculates a checksum for it: D' + C'.
4. The service transforms the data again and calculates a checksum for it: D'' + C''.
5. The service validates the data: C' == C''; D' == D''.
* Steps 3 and 4 should produce the same result. This ensures that there were no bit flips and that the logic is deterministic.
6. The service recalculates the checksum for the original data and verifies its correctness: checksum(D) == C.
* This confirms that all previous calculations were performed on correct data and that there were no bit flips after step 2.
7. The transformed data is sent on: D' + C'.

The moral of the story:
1. You can't blindly trust data received from external services.
2. With a large number of servers, the probability of hardware failure increases.
3. Using checksums increases the reliability of your data.

Have you encountered bit flips in your work? Do you use checksums?

#work #programming #bugs