I’m currently working on a pretty deep optimization cycle in how RavenDB and Voron (its storage engine) handle I/O. The idea is to be able to squeeze more performance out of the system by being able to utilize a lot of moving pieces at once (parallelizing compute, durable I/O, async I/O, etc.).

At the end of the day, however, it all must reside on disk, of course. That means that I’m essentially seeing how close I can dance to the edge of the capabilities of the system. For example, let’s consider AWS GP3 drives. They offer 3K IOPS and 125MB/sec on the basic plan.

We have 50 clients, each generating a request every 15 - 20 ms and writing ~32KB each. This is the kind of load that they’ll generate on the system. Here is what this looks like:

Now, let’s add just 3 additional clients to the mix, add just enough to hit the limit (not even grossly exceed it!). Here is what this looks like:

Note that in very short order, we moved very quickly from < 2ms to > 250 ms. That is the kind of dangerous tipping point that causes the system to just fall over and die.

As a database engine builder, I have to be really conscious about those details, since it is better for us to start rejecting or slowing requests before we saturate the disk like that. When this happens, you often can end up with a system that is simply unable to perform any I/O.

When that happens, you start using more memory (and buffers). When you run out of memory, the OS will decide to try to swap to disk. But the disk is busy. I/O starvation like that can turn into “the entire machine just locked up and doesn’t respond to anything” in very short order.

I have shown you AWS GP3 for a reason; it has consistent behavior. But there are other models. AWS GP2, Azure’s Premium SSD v1, and GCP’s Standard PD all have a different mechanism.

You get burstable I/O - if you need to copy a large file, for example, the system will let you exceed those limits (for a while). You get a certain allowance that allows you to smooth over jumps in usage. That can be very nice, until you realize that this is what this does to the system:

I strongly dislike those sorts of disks, because there is very little that you can do to actually properly prepare for these scenarios. At one moment, you are running smoothly, and suddenly all your I/O is yanked out of your hands. You don’t have time to apply back pressure, for example. It’s like going for a jog and at the 800m line you are wearing lead shoes.

As they say, as a result: hilarity ensues.

All of this exposition is to discuss the details of what I’m trying to do. The work we are doing is to try to make the most of the hardware capabilities we have, and at the same time, be able to apply back pressure on our clients so we can properly perform at high scale.

I’m doing this work with Claude Fable 5. At the time of this writing, this is one of the top frontier models. I find it quite an interesting experience, to be honest. It is like working with an idiot savant, with an emphasis on both parts at once.

It can be incredibly capable of wedging itself into weird shapes. The reason for this post is that I’m currently waiting for a benchmark run to complete and contemplating this statement that it based a whole design on:

bandwidth = maximum size/duration ever observed (a flood cannot make a single write exceed the device's real throughput, so congestion can't inflate this either).

As you can see, this assumption (which was “load bearing” for the entire design) is quite wrong. It is also the sort of thing that caused me to have a jump scare moment when I saw that. No, we do not make these sorts of assumptions here. Don’t ask me why; I got the scars from that.

The interesting part of this assumption is that it is entirely reasonable to make. If I heard that statement from a developer, I would correct that, but it would make sense why they think that.

For a frontier model (that is currently busy doing this exact performance work, I should add), that was surprising. It showed how a small assumption can generate a pretty bad result. To be fair, I had my own similar scenario (which took ~10 years to fix).

I can say that you can expect some nice goodies out of this work.