Let me share another interesting story from the world of software development. This time, I want to tell you how I managed to increase the performance of one of our services by 20 times by tweaking Linux parameters.

One of the related services was facing a problem: after the host was restarted, there was a slight decrease in availability and a small increase in API latency. Several developers had tried to solve this problem for the past two years, but without achieving any concrete results.

A key characteristic of this service is its high load. The peak request rate (RPS) reaches 500,000 on a single host, with an average of around 300,000. Under these conditions, ensuring stable and predictable operation of the network stack is critical. Any delay in packet processing immediately affects other parallel requests.

To solve this problem, I first analyzed the CPU core utilization under maximum load using htop. To my surprise, I found that several cores were overloaded, while most of the others were idle. Interestingly, the same pattern was repeated after the restart, and it became even more pronounced.

I began to suspect that the same core that received the request was handling all the request processing. And if there are too many requests, a queue forms, and the core cannot process everything in time. Given that the requests are relatively small and fast (average latency < 10 microseconds), the scheduler doesn't even attempt to load other cores.

Immediately, a potential solution to the problem came to mind: to move request processing from the cores that receive requests from the network card. This would allow these cores to focus solely on tasks related to the network stack, avoiding the execution of user code on them.

I achieved this in the following way:
1. I configured SMP IRQ affinity, dedicating specific CPU cores to handle network requests. One core per network card.
2. I made sure that Receive Packet Steering was disabled. In essence, this technology is designed to determine which CPU cores should process packets. However, unfortunately, it doesn't guarantee that processing won't occur on the network cores.
3. I configured CPU affinity for the service process using taskset, excluding the cores from step 1. This way, the application could only run on specific cores, forcing the scheduler to schedule request processing on other cores.

This solution resulted in a more even distribution of the load across all cores. The API p99.9 latency decreased from 120 microseconds to 6 microseconds. The application itself is now capable of handling 500,000 requests per second with an availability of 99.95% (compared to the previous 93%). Furthermore, this change will reduce the cost of supporting the service in the long run, as it will be possible to use less powerful hosts.

The moral of the story:
1. Performance problems are not always in the application code. They can also be due to host configuration.
2. When investigating performance issues, it's always worth checking how evenly the load is distributed across the CPU cores.
3. Sometimes, proper host configuration can save a company millions in support costs.

#work #programming