Tldr:

1. A robust storage engine should remain stable during heavy workloads.
2. The current RocksDB delayed write mechanism based on heuristics can cause unstable applications and stalls under heavy workloads.
3. Speedb has developed a new delayed write mechanism to improve RocksDB stability, that considers user-defined values and changes the write rate in a more stable, linear way.
4. This new mechanism enhances application performance stability over time by slowing down or speeding up as needed.

Not everyone at the same time

Imagine you have a busy day ahead, with tasks like taking your kid to a piano lesson, completing a code review for your teammate, fixing critical bugs, and still wanting to watch the world cup with friends in the evening. With so much on your plate, it can feel overwhelming to manage everything.

Similarly, applications can become overloaded and unresponsive when they receive too many requests simultaneously. However, having a mechanism that can control the number of requests a key-value store receives can improve performance and provide consistency over time. This helps to make your application more predictable and reduces the likelihood of frequent stalls or idle situations. Developers should prioritize implementing such a mechanism to optimize application performance and ensure a better user experience.

Speedb is committed to building a more robust storage engine, and one of our main objectives is to stabilize the performance of existing key-value store (KVS) solutions. A critical component of KVS performance is the delayed write mechanism, which controls the rate of writes the system allows the user to send

Existing RocksDB delayed write mechanism

RocksDB delayed write mechanism is based on the latest system state.  By monitoring three parameters - compaction pending bytes, number of files in L0, and if the last memtable is being used - the mechanism decides if to trigger the delayed write mechanism. If so, the system increases or decreases the write rate by the same amount, based on the last system state, This can cause the delayed write rate to change dramatically even with a small change in the compaction bytes, and the write rate to decrease significantly when the initial write rate limit is high. This can lead to long stalls when the system is dealing with a heavy workload, causing the application to idle while waiting for the KVS to catch up.

To avoid these issues and provide a more stable and predictable application performance, developers can use more dynamic and adaptive delayed write mechanisms that can adjust to a limit defined by the user, in a more gradual and accurate manner. By doing so, they can optimize KVS performance, make applications more stable and predictable, and provide a better application experience.

‍

Speedb’s improved delayed write mechanism

To simplify the code and improve performance, we have made some significant changes to the delayed write mechanism. Rather than basing the calculation on the previously calculated state, we now take into account user-defined values. This approach helps to ensure that the write rate is controlled moderately, which helps stabilize application performance over time.

If we compare this to a driving car that needs to stop at a red light, the heuristics-based mechanism first presses hard on the brakes and then keeps pressing moderately. Alternatively, the new mechanism will gradually slow down the car with no dramatic changes, so the passengers won’t feel uncomfortable.

‍

Delayed write benchmark RocksDB and Speedb

To illustrate the benefits of the new delayed write mechanism, let's consider an example. Suppose a user has set a delayed write rate of 100MB/s, with a soft limit of 10 L0 files and a hard limit of 20 L0 files. The new mechanism will adjust the write rate gradually based on the current number of L0 files, rather than abruptly slowing down or stopping the system. This approach helps to avoid sudden or significant changes that could destabilize application performance and cause stalls or other issues. By implementing more gradual and adaptive write mechanisms like this, developers can improve the stability and predictability of their applications over time, even under heavy workloads. 

The example below- we only use a single parameter as a baseline for the decision rather than all the three parameters mentioned earlier

User-defined delayed write rate =100MB/s

Number of L0 files to trigger slow down (soft limit) = 10

Number of L0 files to trigger stop (hard limit) = 20

‍

‍

The table above shows the write rate changes that occur with the new delayed write mechanism. Compared to the previous RocksDB delayed write,  the new Speedb mechanism adjusts the write rate in a more gradual and adaptive way that is better suited to the system's current state. It slows down when necessary and speeds up when it can, providing a more stable and predictable performance for the system. By adjusting the write rate in this more moderate manner, the new mechanism helps to avoid sudden and drastic changes that can lead to stalls or other issues, resulting in improved performance and a better user experience.

The graph below, “Seek random while write”, shows a comparison between the old and the new mechanism, using the db_bench tool.

The graph compares the performance of the old flushing mechanism used with RocksDB and the new mechanism used with Speedb when running an application on top of each respective database. The graph shows that as the number of operations per second (OPS) increases, the RocksDB configuration experiences frequent stalls, while the Speedb configuration is able to maintain a high level of OPS with only one stall throughout the entire run.

Summary 

Comparing the new delayed write mechanisms with the old mechanism reveals that the heuristics-based mechanism can lead to a significant decrease in the write rate, even with a slight change in the state. This can cause the system to experience frequent stalls and result in unpredictable and unstable application performance. The decrease in write rate is particularly noticeable when the initial write rate limit is high. In contrast, the new mechanism adjusts the write rate in a more gradual and adaptive way that is better suited to the system's current state, slowing down or speeding up as needed. This results in more stable and predictable application performance, even under heavy workloads.
‍

The new delayed write mechanism is available in Speedb v2.2.0. Check it out!  https://github.com/speedb-io/speedb

Appreciate what Speedb is building?  Star us on GitHub,
We are waiting to get your feedback!

Release: https://github.com/speedb-io/speedb/releases/tag/speedb%2Fv2.2.0 

‍