In my last post on the topic, I talked about ImmutableDictionary and why it was too costly for us to use. At the same time, it is basically the data structure that we need for the task at hand. That task is keeping track of the current set of modified pages, with multiple concurrent versions running at the same time.

ImmutableDictionary is perfect for the task, because it gives us the right immutable behavior, but its performance makes it unacceptable. So the solution is to design a data structure that would fit our needs and requirements.

The people who wrote ImmutableDictionary are not stupid. They didn’t set out to make something that would be slow. The issue is that they set out to solve a generic problem. We can benefit from specializing the data structure for our needs (removing a lot of complexity along the way).

Concretely, we need something that:

• Provides O(1) lookup.
• Plays nicely with the GC and allocates rarely.
• Has a simple structure for the GC to work with (not a lot of interior pointers).
• Still supports the snapshot model that made the immutable dictionary so attractive: the ability to discard values while old snapshots remain valid and consistent.

The problem is combining all of those together. It is much easier if we can drop any of those requirements.

I started the design by dropping the very notion that this is a generic data structure. It isn’t an ImmutableDictionary<TKey, TValue>; instead, we have the ScratchPagesTable data structure.

Conceptually, this is similar to a Dictionary<long, ScratchEntry> in terms of how we’ll use it. ScratchPagesTable is essentially an open-addressing hash table, using linear probing and Fibonacci hashing. That is pretty basic, to be honest. That isn’t what makes it interesting.

Here is the actual layout of the data structure:

The entire design for this table is meant to make it easier for the GC to work. Note that the first three arrays here are plain objects. The _keys and _heads are parallel arrays, which we use to find the actual entry in the _entries array.

The _entries array contains the ScratchEntry structs, which you can see on the right side. Each struct contains all the details about the current entry. Note that the entry also does not use any pointers.

The only pointers in the entire ScratchPagesTable are in the _refs array. Each ScratchEntry in the table points to a location in a particular file. Instead of carrying a pointer in each of the (many) entries in the table, we put them all in the _refs array and refer to them by index from the entries.

We may have tens to hundreds of thousands of entries in the table, but we’ll usually have a much smaller number of files (almost always under ten). By centralizing the references in a single small array, we made the job of the GC so much simpler.

When it needs to scan the ScratchPagesTable to see what is used, it can mark each of the top three arrays immediately and not walk their contents (because they have no pointers). And the _refs array is very small, so that isn’t meaningful.

Let’s talk about the concurrency aspect here. We have the ability to take cheap snapshots from this data structure, which gives us a stable view of the data.

We take advantage of the fact that we have a single writer, so there is just a single thread that mutates the table. Then that same thread is the one that is publishing the new snapshots to other threads.

Here is what this looks like:

A snapshot simply captures references to all four arrays, the total count of items in the table, and, most importantly, the sequence number that was stamped on the snapshot when it was created.

If you look at the ScratchEntry layout, you’ll note that we have the OlderIndex field and the Seq field. Together, those represent a linked list that we can scan to find the first entry (value or tombstone) whose sequence number is less than or equal to the current sequence number that we are using.

A reader using a snapshot holds a mutable array, but the use of the sequence number ensures that it can never actually observe changes that happened after the snapshot was taken.

We don’t need to worry about concurrent mutation, since this is meant to be used with a single publisher (already guaranteed by the way RavenDB works).

Of course, if we just keep on using the table in this manner, we’ll have to deal with garbage. How do we get rid of old entries that no snapshot looks at? Don’t we have to replace the GC ourselves?

Let’s go back to the concept of the sequence. RavenDB keeps track of all the active transactions, and each of those holds a snapshot. That means that we have a cheap way of getting the oldest snapshot and its sequence.

Whenever we create a new write transaction, we check what the oldest sequence number is. That means that when we update the scratch table, we can prune the chains when we find entries that no one is able to look at any longer.

Pruning the chain in this manner doesn’t mean that we can immediately use the entries we unlinked. The issue is that a reader may want to read them. Let’s assume we have a deletion, like in the diagram above. When we prune, we cannot just immediately reuse the entries, because the readers may have already started looking at them.

What we do instead is park them, as you can see in the following diagram:

Once we have progressed through enough transactions and all currently opened transactions started after the entries were parked, we can reuse those entries. Of course, if we have enough entries, we may need to allocate brand new arrays (re-hashing). That is safe to do because all the old snapshots contain the old references; we just discard them.

This model of pruning the old versions from the end is great for most scenarios. However, there is a particularly nasty use case where it utterly breaks down. What happens when we have a long-running transaction? We have to keep all the versions alive until the oldest read transaction is done with them.

As it turns out, there is a very common scenario where this can happen. If we run a backup, we may hold a read transaction for a long time. Write transactions while the backup is running will cause the chain to become longer and longer. The design of the scratch table means that we keep things fast for the current transactions and recent ones.

The old transaction has to traverse the entire chain each time, becoming slower the more time goes by. The slower it becomes, the more time it takes, the more write transactions can happen, the slower it becomes… That isn’t a fun scenario.

RavenDB handles this scenario explicitly. When we detect that the chain is too deep, we start pruning the chain. We discard all the intermediate transactional versions that exist between the current committed version and the oldest read transaction.

You can see what this looks like in the following diagram:

Using this mechanism, we can keep the amount of work that old transactions need to do under control.

The end result of all of this work is a high-performance concurrent data structure that takes into account both how it is actually being used and the internal intricacies of the .NET garbage collection. It isn’t your run-of-the-mill data structure, nothing that would be relevant off the shelf.

It is also directly responsible for an improvement of over 20% in the performance of RavenDB when we compare the two versions side by side.

Over a quarter of a century ago, Joel Spolsky wrote about Netscape’s decision to rewrite its browser from scratch, which later led to Firefox. He called it Things You Should Never Do, Part I.

I remember reading that article at the time and fully agreeing with him. As a side note, it is well worth the time to go through the archive of his posts. Even today, a lot of them hold up very well.

Reading it back again, I found the following quote, which is really appropriate today:

Before Borland’s new spreadsheet for Windows shipped, Philippe Kahn, the colorful founder of Borland, was quoted a lot in the press bragging about how Quattro Pro would be much better than Microsoft Excel, because it was written from scratch. All new source code! As if source code rusted.

To be clear, he was writing in (April 2000) about something that happened in 1992, so 34 years ago. That statement is still very much in force today. Another quote from this article that still lives in my head is:

Back to that two page function. Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

There is an interesting question here. Do we have the same problem with AI? After all, AI can write a lot of code. Does it matter?

This post was started when I was working on optimizing RavenDB. That is something that I do as a habit, in an ongoing process. Right now I’m looking into optimizing the scenario of how RavenDB starts up after a cold boot, when running on an I/O-constrained system.

The scenario is a 100GB database running on an AWS gp3 disk with 125MB/sec and 3K IOPS. In most cases, you size up your machines so the working set will fit comfortably in memory. But what about when you start up after a cold boot (memory is empty)?

In that case, the access pattern you have for the data is really important. That is also not something that you’ll typically see in “I used AI to generate a storage engine”. That is the sort of “detail work” that is not interesting to teach (no fancy coding here); it isn’t “my benchmark got 15% faster” (it is very specific to a single scenario).

In long-lived software, however, you accumulate such features. Over time, those end up being a very significant advantage.

Another gem from Spolsky is this blog post, where he talks about software bloat, which is the other side. Then he points out that the review for a new word processor pointed out that the software is very fast, but it lacked (for a journalist) an essential word count feature.

Can you get AI to write all those features and behaviors? Of course, but in order to actually run into them (so you know what to do), you need actual in-field experience.

To really hammer that home, you need that support call when they are looking at the downtime clock while the disk is reading in small increments and their business is down. It’s the same as the old story about the chalk mark.

Going back to the original scenario of optimizing RavenDB for I/O-constrained environments, you need to understand how disks work. In general, for a gp3 disk, the cost of reading 8 KB is the same as reading 128 KB. If you are working from a cold boot and try to read an 8 KB page from the disk, it actually makes sense to read more to try to get things into memory faster.

That is where things get interesting, because the previous statement is true only about IOPS; you also have to balance bandwidth limits, as well as what happens when your memory is full and you’ll need to evict data from the cache.

RavenDB is using prefetching; it will analyze patterns in how you access the data and proactively try to read more when it can. That can dramatically improve cold boot performance.  

Over a decade ago I tested immutable data structures inside RavenDB. The verdict at the time was simple: the performance was unacceptable. That was a long time ago, and one of the things I decided to do recently was to revisit all of it. I looked into that, and the situation didn’t change.

The reason this matters is a piece of code inside of RavenDB that is called the Page Translation Table. It is how RavenDB handles MVCC (transactions, isolation, etc.), and to call it critical for performance would be a massive understatement.

The Page Translation Table holds the set of pages that have been modified by a transaction. You may have multiple outstanding transactions running at the same time, and when you want to read a particular value from the system, you have to go over all of the currently outstanding transactions and check whether that value is one of the temporary pages they modified, or if that value is on the disk.

RavenDB’s current PageTable is based on a ConcurrentDictionary holding a versioned view of the pages (with each page version associated with the relevant transaction). If the description sounds complex, you are correct, it is. It was also one of those things that significantly complicated some major re-architecting that we wanted to do.

As part of a major refactor of RavenDB's internals, we moved the data out of the existing page translation table and into something more natural: an immutable dictionary. More specifically, the BCL’s ImmutableDictionary class.

Note that this is the second or third time that we have come back to ImmutableDictionary as the data structure of choice for this particular purpose. In pretty much all respects, that is the perfect data structure for what we want. It lets us capture a snapshot of all the modified pages inside a transaction, hold onto that snapshot cheaply and easily, and stop worrying about concurrency and race conditions entirely.

If another transaction modifies things, that simply does not affect the snapshot I am holding.

That is a great building block for the MVCC infrastructure that we need inside of RavenDB.

The reason we keep not using ImmutableDictionary is pretty simple, however. The performance is just not there for our needs.

As part of the new architecture, we threw away the old page table, implemented the new architecture (and reaped all the benefits thereof) using an ImmutableDictionary and set out to test what this looks like. It was really bad.

When we dug into the details, the problem child was the ImmutableDictionary. So let us talk about how it actually works. While the ImmutableDictionary has the same interface as a normal dictionary, it is actually implemented as a tree internally.

Every time you mutate it, you create a new node in the tree. That is exactly how it manages to be immutable, and that is exactly what makes it such a pleasure to work with under concurrency. Now consider the impact of that in terms of performance.

The first cost: lookups are no longer O(1)

Say I have a dictionary with ten thousand items. What is the expected cost of a lookup? If this is a dictionary, I expect O(1). Every assumption inside RavenDB is built on the premise that getting this value costs O(1). Since this is a tree, the cost of a lookup is O(log N). For 10K items, the cost would be 14 pointer-chasing operations.

Note the emphasis on pointer chasing? That is a huge cost to consider. Especially for something that runs literally billions of times a second.

The second cost: allocations per mutation

Consider Dictionary<long,IntPtr>, which is what the Page Translation Table in RavenDB boils down to. Then consider it as an ImmutableDictionary. Using a standard .NET dictionary, you get essentially no allocations for writing to the dictionary: all the data consists of value types, stored inside arrays with amortized allocations.

In an immutable dictionary, every single one of those entries is an allocated node. What is worse, the more items you have in the dictionary, the bigger the tree, the more allocations you have. Ouch!

To be clear, if you have a dictionary with tens of thousands of items in it, you’ll allocate more than a kilobyte for each mutation. If you have to do 100 updates in a dictionary with 100K items, you’ll allocate more than 100KB for that.

The third cost: the GC has to walk the graph

The internal structure of an ImmutableDictionary is a tree of references. And that is a pretty complex object graph. The GC needs to walk that entire graph (pointer chasing all the way), and the more complex the object graph, the more work the GC has to do. That is enormously wasteful.

Now put that under heavy load, meaning a lot of items going in and out of the dictionary. The structure is producing many allocations and snapshots as it works. With a high churn rate of values, an ImmutableDictionary is incredibly inefficient in terms of GC cost.

So we end up paying for the ImmutableDictionary in three separate ways: the O(log N) lookup, allocations per mutation, and the GC time spent walking the object graph.

Using the same 100K example as before, here is what this looks like. Note the number of pointers that you have and the pointer chasing that this requires from the GC.

This GC cost is nasty, because it does not show up as a single hotspot in the profiler. It shows up as a smear of additional CPU usage spread across the whole system.

Summary

For the third time, we had to discard the option of using ImmutableDictionary because the performance cost was simply unacceptable. We still gained a lot because using it as a model for the re-architecture let us land on a much better design.

We needed a better option. In my next post, I will get into the specifics of how we actually built that solution.

I was recently asked to look at a change one of our developers made to RavenDB. He was chasing a particularly slow query with spatial sorting (a “find me all the nearest Thai restaurants” style of query). He found the root cause and had a fix ready. The fix gave us a 25% improvement in query processing time, and the change was about 15 lines of code.

A few years ago, I would have said: great job, merge it, ship it, let customers enjoy the faster queries. But I got a hunch that we could do something different. We asked the AI model to try it out. It took a non-trivial amount of work to even test that, while in the past, it would have taken a few days just to try it out.

The model was able to complete the spike while we were talking about why this should be a good approach and the architectural-level aspects that were driving it.

The result of that spike was an improvement of 12x. Not 12%. Twelve times faster. A 25% improvement is a wow;a 1,100% improvement is damn. In essence, we took a shortcut a couple of decades ago and had to pay a lot more than we could have.

Note that a 15-line fix is local, easy to understand and verify. A structural change like this is… not. We also have to deal with backward compatibility concerns (we had to change the index format), verify that we get the same output in both cases, etc.

Along the way, we found three separate bugs that had been in the product for 10 to 15 years. They're all in very weird edge cases. What happens when you sort on a spatial field that doesn't exist?

If a document has no spatial value, we would treat the missing value as zero distance, which means it is the closest result. I can argue that this is the correct behavior, but not convincingly.

The PR went from 15 lines of code to about a thousand. Review time for that went from under 10 minutes to over an hour.

Compare this to the same task four years ago: a few hours to isolate and understand the problem, a couple of hours to figure out how we could optimize this, write the code, etc. The overhead of actually reviewing the code and merging it would be pretty minimal.

Today, it took an hour to reproduce the problem, another with the model to find & propose the original 15-line fix. Then we dove deeper into this and did the rest in another couple of hours. The 1K LoC PR would take two hours to review.

Interestingly enough, I don’t think we would have hit on the 15-line fix if we had been working manually. That took advantage of a library function I didn’t even know existed. We would have headed directly to the proper fix we ended up with.

That leads to a few observations about the process. The quick fix with the AI would have delivered an inferior end result. It is funny when I say that a 25% improvement is inferior, but it would have been in this case.

The other aspect is that each decision we made along the way made sense, but we ended up with something that was big. To the point that you need to meditate over the code to understand it.

This is a very small and focused change, mind. And it grew from 15 lines to 1K. We are seeing the size of PRs grow from a few hundred to many thousands of lines of code.

All of that pushes the complexity into the review process. And for long-lived software, review is not something you can skimp on. RavenDB has been around since 2009. Code we merge today is code we will be supporting and reasoning about for the next decade.

There is a wide spectrum of opinions on this right now. Some people say they never read the code; some have an agent review it. Others insist on going through every line, and that is where I stand.

The code that goes into RavenDB gets a manual review by a person who understands everything about the system. Not just whether the code is correct, but how it interacts with all the other stuff we have, the history behind it, and the things that are not in the diff.

Productivity? Where is it at?

As an industry, we expected AI to make us dramatically faster. If AI writes 90% of the code, sometimes 100%, shouldn’t we be shipping more often? And better software as well, no?

Every developer I've talked to says the same thing: I cannot not use AI. The benefits are too high to give up. I have seen people who left the IDE entirely and some who just use AI to explain and reason about the code, but everyone is using AI in some capacity. But we aren't seeing the change in productivity that everyone predicted.

I think a big part of the answer is in the story above. We aren't doing the same work faster. We're doing more work, arguably better work. We find opportunities we would never have considered, because the cost of checking a hunch dropped significantly. The cost of accepting the answer didn't decrease; in fact, I would argue that it became much higher.

To be clear about the scale of this: RavenDB did not suddenly get 12 times faster across the board. As much as I wish a 1K PR could do that. A single scenario got dramatically better. That is great, but it is also more code, more behavior to understand, and more surface area to maintain.

When we keep going like that, we end up with better software overall, but also a much larger mass of code to work with. It also means much more effort from the people who actually have to deliver it.

I think that we’ll see major shifts in how we build and maintain software, but at the same time, I think that the key fundamentals haven’t really changed. In the past quarter, RavenDB (the company, not the software) hired more R&D staff than we did in the past two years, to give you some context.

There is going to be a greater need for developers than ever, and even more digitalization of the world.

I started RavenDB in a spare bedroom, because of a problem that wouldn't let me sleep. Everywhere I looked, applications were fighting their own database. ORMs papering over a model that didn't fit. Indexes missing or just bolted on after production failures, without attention or concern to how to do this properly.

That was the common situation in the industry; accepted because databases required a lot of expertise and no one had time to do the right thing with the next release date looming. The industry's answer was always another layer of tooling on top of a design that was already wrong. So I built a database where those things were expected, that would behave properly even if it didn’t have a full time babysitter to make sure it survived production.

Twenty years later, I'm looking at the same shape of problem again.

The same feeling, a second time

In the past couple of years, we built RavenDB's AI capabilities and we've spent the last year helping our customers launch complex AI initiatives. Not just the sorts that make for a cute demo. Stuff that runs in production and intelligently serves hundreds of thousands of users. The kind that has to work in the field, in front of real users, and just keep ticking.

Looking at the overall AI landscape, I think we, as an industry, have a problem. I have not seen this much effort spent on bringing so little value since 2008. That is the sort of thing that makes my fingers itch to go ahead and fix it.

The numbers say the same thing. Demos are easy, and a weekend gets you something that impresses the room. Getting that same thing into production takes 12 to 18 months of plumbing work and endless testing, and out of a hundred initiatives, about five make it.

“Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls”

Most enterprises have their data sitting in existing systems and databases. All of them have been running for a decade or more. The entire company relies on those systems, very few people actually understand how they work and their internal interactions. Nobody is volunteering to touch them. The phrase “there be dragons” was coined specifically for those kinds of situations.

At the same time, there is an intense pressure from the market and management to introduce AI in all things. That typically means that you need to bring in new AI-ready infrastructure. Modifying the core systems of the business is tricky. No wonder that those projects take so long, or fail so often.

Remember, it's not just one system either. Working out a customer's status at a bank means the checking balance from one system, the mortgage from another, the credit card activity from a third. You also have to consider the interactions between all those systems. To bring that to the world of AI is a huge leap.

The gap you have to cross is beyond just throwing data at a Large Language Model. It's a data organization, understanding, and governance one. You have to first get your house in order before you can take the first real step.

So we built Quill

Quill is an AI Context Layer. It sits between your existing databases you already run and it provides the AI infrastructure that you need to build sophisticated AI capabilities in your organization.

Quill takes upon itself to do the work most teams are currently doing by hand.

A typical AI project requires you to first get your data governance story in order, understand how to effectively utilize Large Language Models and keep chasing the latest innovations. That previous statement, translated to calendar time, is about two years. In work-years, it is usually measured in multiple decades.

Quill changes this in a dramatic fashion, here is how it works, at a high level:

1. Point Quill at your existing databases (Postgres, SQL Server, MariaDB, MySQL).
2. Tell Quill what aspects you are interested in, and it will deal with the minutia of turning your existing data into the proper shape for AI to consume and work with.
3. Describe what sort of AI capabilities you want (AI bank teller, for example), and Quill will generate that for you.

What is actually happening behind the scenes is that Quill is pulling and transforming the data from your existing databases into RavenDB. This is done on a continual basis, so the data is always up to date. Once the data is inside RavenDB, you can then apply all of RavenDB’s own AI capabilities. Such as vector search, embedding generating, Generative AI and AI Agents.

Building on top of the proven RavenDB infrastructure, you get to skip a lot of the hassles in properly delivering an AI project. Details like Bring Your Own Model, audit trails, rate limits, vector indexes and more.

Most importantly, Quill allows you to remain in control at all times. Your data never leave your servers, you don’t have to trust a cloud service with your most precious assets. You can control what information will be available to the AI model. Finally, both RavenDB and Quill take the approach that the AI model is not a trusted entity.

Typical AI projects use MCP connectors to expose your system to the AI model. That puts the AI itself in the driver’s seat. Quill takes the opposite approach, the AI is there to facilitate and provide the intelligence. It is never in control, and it is intentionally limited in what it can see and do.

Quill takes the notion of security much more seriously than merely using row-level permissions. Any interaction with the model is scoped to seeing just the information that the model should see in its current role as an agent of the user.

Quill Design Philosophy

Quill assumes that the AI model is inherently vulnerable. Quill’s approach is to limit the scope of what the AI can access and the actions it can perform. The AI is considered to be an agent of the user, not the organization.

If a user manages to prompt-engineer the AI agent, they are still behind the same security door, with no privileged access possible.

Quill’s AI capabilities allows you to immediately deploy agents in your application and in channels such as Telegram, Slack, or Discord.

What you end up with

Agents working from data that always is up to date, seeing only what they're cleared to see, running on your own servers. That's the whole thing.

The idea with Quill is that you can rapidly AI enable your systems in a safe and reliable manner. Instead of taking years to deliver AI projects, you can be in production in mere weeks.

More importantly, Quill isn’t built to be just a nice demo. Quill was built to be the AI infrastructure that you need to scale your AI in as they grow in complexity and scale easily with a huge number of users.

Quill is built on top of RavenDB’s proven AI infrastructure. Running AI agents in production, serving hundreds of thousands of users in a highly regulated industry.

I spoke earlier about the ease of onboarding. Just point Quill to your existing databases, click “Next” a few times and you have a running AI agent that you can immediately start working with. Quill matters because it is there to handle all of the logistics around that, and all the “-ities” that you need to concern yourself with.

From built-in memory so your agents retain continuity between interactions, to audit trail so you can see past conversation, handling compaction, guardrails, mutli-agents and multi-channel conversations.

Quill allows you to build a simple AI assistant that can access the relevant data to answer questions. You can then expand on that to build sophisticated AI agents that take actions on behalf of the user.

You have the space and ability to grow your AI capabilities in an iterative manner, without having to invest several work-years into a project with high probability of failure. We are confident that you’ll be able to go from zero to production deployed AI capabilities within weeks. Quill is offered with a 3 months unlimited free trial.

Quill is generally available today. Go build your first agent now.

I’m currently working on a pretty deep optimization cycle in how RavenDB and Voron (its storage engine) handle I/O. The idea is to be able to squeeze more performance out of the system by being able to utilize a lot of moving pieces at once (parallelizing compute, durable I/O, async I/O, etc.).

At the end of the day, however, it all must reside on disk, of course. That means that I’m essentially seeing how close I can dance to the edge of the capabilities of the system. For example, let’s consider AWS GP3 drives. They offer 3K IOPS and 125MB/sec on the basic plan.

We have 50 clients, each generating a request every 15 - 20 ms and writing ~32KB each. This is the kind of load that they’ll generate on the system. Here is what this looks like:

Now, let’s add just 3 additional clients to the mix, add just enough to hit the limit (not even grossly exceed it!). Here is what this looks like:

Note that in very short order, we moved very quickly from < 2ms to > 250 ms. That is the kind of dangerous tipping point that causes the system to just fall over and die.

As a database engine builder, I have to be really conscious about those details, since it is better for us to start rejecting or slowing requests before we saturate the disk like that. When this happens, you often can end up with a system that is simply unable to perform any I/O.

When that happens, you start using more memory (and buffers). When you run out of memory, the OS will decide to try to swap to disk. But the disk is busy. I/O starvation like that can turn into “the entire machine just locked up and doesn’t respond to anything” in very short order.

I have shown you AWS GP3 for a reason; it has consistent behavior. But there are other models. AWS GP2, Azure’s Premium SSD v1, and GCP’s Standard PD all have a different mechanism.

You get burstable I/O - if you need to copy a large file, for example, the system will let you exceed those limits (for a while). You get a certain allowance that allows you to smooth over jumps in usage. That can be very nice, until you realize that this is what this does to the system:

I strongly dislike those sorts of disks, because there is very little that you can do to actually properly prepare for these scenarios. At one moment, you are running smoothly, and suddenly all your I/O is yanked out of your hands. You don’t have time to apply back pressure, for example. It’s like going for a jog and at the 800m line you are wearing lead shoes.

As they say, as a result: hilarity ensues.

All of this exposition is to discuss the details of what I’m trying to do. The work we are doing is to try to make the most of the hardware capabilities we have, and at the same time, be able to apply back pressure on our clients so we can properly perform at high scale.

I’m doing this work with Claude Fable 5. At the time of this writing, this is one of the top frontier models. I find it quite an interesting experience, to be honest. It is like working with an idiot savant, with an emphasis on both parts at once.

It can be incredibly capable of wedging itself into weird shapes. The reason for this post is that I’m currently waiting for a benchmark run to complete and contemplating this statement that it based a whole design on:

bandwidth = maximum size/duration ever observed (a flood cannot make a single write exceed the device's real throughput, so congestion can't inflate this either).

As you can see, this assumption (which was “load bearing” for the entire design) is quite wrong. It is also the sort of thing that caused me to have a jump scare moment when I saw that. No, we do not make these sorts of assumptions here. Don’t ask me why; I got the scars from that.

The interesting part of this assumption is that it is entirely reasonable to make. If I heard that statement from a developer, I would correct that, but it would make sense why they think that.

For a frontier model (that is currently busy doing this exact performance work, I should add), that was surprising. It showed how a small assumption can generate a pretty bad result. To be fair, I had my own similar scenario (which took ~10 years to fix).

I can say that you can expect some nice goodies out of this work.

A while ago, MongoDB purchased VoyageAI for 220 million dollars. Since then, they have released a couple of dedicated embedding models. For example, you can read their blog post on voyage-context-4.

The key premise in those sorts of models is that you can feed the model text of any size, and it will automatically handle generating embedding vectors, smart chunking, providing context, etc.

I ran into this recently and was curious to see how this can work. In particular, since RavenDB handles both embedding generation and vector search, I decided to do a full evaluation of MongoDB’s way of chunking. MongoDB built their own model to achieve this, but RavenDB’s approach to embedding generation is to rely on any embedding model you prefer to use.

Before we get into the full details, let’s talk for a second about what the point of contextual embedding is, so we are all on the same page.

Embedding models take your data and translate it into a multidimensional mathematical space based on its meaning. Similar items will be located near one another in this multidimensional space, and we can take advantage of that using vector search. That is why you can find Mozzarella & Ravioli if you want Italian food today, as in this example:

The problem is that all embedding models have a context limit. There is only so much text that you can push into the model before it will give up on you. If you want to search through a much bigger piece of text using semantic search, you need a different approach.

The industry standard approach to handling this is via chunking. In other words, you take a long piece of text, split it into separate parts called chunks, and generate an embedding for each one separately.

The easiest way to think about this is that you have a long document, and you generate a separate embedding vector for each page of text independently. Instead of having to digest a whole article, you feed a bounded chunk (page) to the model to generate an embedding vector.

Chunking is a neat trick, but it leads to its own set of problems. Assuming we have a large document that talks about new features in RavenDB, with a particular page that expounds on the details of “the database’s ACID guarantees". What would the embedding vector for that page look like?

If we just chunk the data naively, we’ll get a vector that is related to the generic concept of ACID in databases. The chunking approach loses the context of the data; it doesn’t understand that the database in question is RavenDB.

Contextual embedding allows you to bake a global perspective directly into every chunk’s embedding. In other words, the embedding for that page would know that the database that is being talked about is RavenDB.

If you are dealing with large texts and want to have high-quality search, contextual embedding is a feature you want. I guess that explains why MongoDB paid 220 million dollars for Voyage AI.

Sadly, I left that sum of money in my other pants, so RavenDB’s strategy for dealing with this scenario is quite different. We planfor models to become a commodity, so there is little benefit in trying to produce your own models at this point in time.

Instead, RavenDB takes the approach of working with all off-the-shelf models. That means that we are far more flexible, using the latest state-of-the-art models, instead of having to keep chasing them. But only some models support contextual embedding…

Luckily, we figured out that we can add this feature from RavenDB’s side, without needing to develop a custom embedding model for this. The technical announcement about it is here, with all the details. But the gist of it is that RavenDB allows you to attach context to the value you send for embedding.

The scenario below shows an example of storing litigation files using RavenDB and enabling proper semantic search over large amounts of data:

const tokenCount = 2048;
const overlap = 128;


const chunk = (field) => text.splitParagraphs(field, tokenCount, overlap);


embeddings.generate({
  FullDetails: chunk(this.FullDetails),
  CaseSummary: chunk(this.ExtractedSummary),
  PartiesInvolved: chunk(this.MetadataParties),
  Precedents: chunk(this.CitedAuthorities),
  RatioDecidendi: chunk(this.CoreLegalRules),
  ObiterDicta: chunk(this.DissentingArguments)
})
.withContextPrefix(this.Headline);

You can see that we generate embeddings for quite a few fields. For all of them, we use a chunking strategy of 2K tokens with an overlap of 128 tokens. Note the last line that adds a withContextPrefix call, where we add the Headline as part of the context for the data we’ll be embedding.

This additional context gives the embedding model enough information to contextualize the information we give it. The nice thing about this feature in RavenDB is that we don’t need to have any special support from the model. Everything is handled directly by RavenDB. That includes chunking, caching, adding the context, etc.

What to do when I don’t have pre-existing context to add?

If you have a title for an article, or a summary already written for you, that is great. But what happens when no such thing exists? You can also use GenAI tasks in RavenDB to process the data and get a proper summary (and then generate the embedding with that summary to have better queries).

I took the context prefix feature for a spin with a bunch of well-known datasets in the field of embedding and retrieval. We are using nDCG@10 — normalized Discounted Cumulative Gain at rank 10, the standard retrieval-quality metric on the public BEIR, LoCoV1, and LongEmbed benchmarks.

The underlying embedding model we use is OpenAI’s text-embedding-3-small, and we use exact() vector search in RavenDB, since we are testing purely the embedding output.

Adding context to chunked documents

For the following benchmarks, we defined two embedding tasks. One that would simply generate chunked embeddings from the raw text (with 256 tokens per chunk), and another with additional context taken from the document’s title.

embeddings.generate({ 
    ContentEmbedding: text.split(this.Content, 256)
        .withContextPrefix(this.Title) 
});

TREC-COVID        +9.2

COVID-19 literature comprising ~129K near-identical CORD-19 papers with short, keyword-like queries. The abstracts all look alike, so the paper’s title is the single most discriminating signal. You can see that this approach is able to provide better results than any of the other options.

Fair benchmarks are hard (we made it harder for us)

In the following benchmarks, the ravendb and ravendb+ctx entries are the only ones that are actually using chunking. In other words, all the other alternatives are getting the full document to work on. And indeed, you can see that the ravendb entry (which does native chunking) isn’t doing that well in this benchmark. With the added context, it reaches the top.

Chunking at 256 tokens was used because it is a reasonable chunk size (about two paragraphs of text), and at that size, you may lose the context of the overall document. This allows us to showcase how effective the additional context technique is. That is also quite useful for additional focus. Embedding quality degrades with the length of the text, so shorter chunks embed their concepts much more faithfully.

NFCorpus        +2.8

Consumer-health and nutrition queries matched against PubMed documents. Titles name the medical topic (such as a condition or a nutrient), which disambiguates heavily overlapping biomedical text; the prefix lifts us past published text-embedding-3-small results. In fact, only text-embedding-3-large is able to do better than us here (see below for benchmark results showcasing RavenDB’s approach with text-embedding-3-large).

SciFact        +1.1

Scientific-claim verification against research-paper abstracts. The title names the paper’s specific finding, nudging near-duplicate abstracts apart, but the abstract is already on-topic. The gain is modest, and we land within a point of the published te3-small score.

As you can see, in this case ravendb+ctx is doing better than ravendb. However, I wouldn’t say that it is doing well. The chief problem is that chunking to such a small size really hurts us, and just using the full document is better.

Testing additional context with text-embedding-3-large

We intentionally test this approach with a modest model (text-embedding-3-small that has ~100M - 300M parameters). Does this approach scale when we use a bigger model? The text-embedding-3-large model is estimated to be in the 1B - 2B parameter range. How does it behave when we use the same technique?

In the graph below, we are testing the Legal Case Reports dataset, which has a lot of large documents (some with > 100K tokens and many over the 8K token limit).

We tested the quality of the results with chunking of 256 and 4096 tokens.

You can see in the graph that text-embedding-3-large is indeed better than text-embedding-3-small. There is a +3.3 difference between the baseline numbers of both models.

With the context option, however, text-embedding-3-small is almost as good as text-embedding-3-large! And with context, text-embedding-3-large ismuch better.

We also tested text-embedding-3-large with a much larger chunk size of 4K, which should give it more context to draw on (but also dilutes that contextt). Even so, it wasn’t able to beat the additional context (with a much smaller chunk size).

Dealing with large documents

The previous datasets we dealt with all had documents that fit nicely within an embedding model context window. Now we are going to deal with much longer documents (5K–470K tokens each). To make things more interesting, these have no natural title to anchor a chunk.

To handle this, we use another RavenDB AI feature, GenAI Tasks, which reads the first 16K of the document and generates a short summary for it. We then use that summary as the additional context for the chunking.

Without further ado, here are our results:

Those datasets were taken from the LoCoV1 and LongEmbed datasets. They are quite large and are usually used to explicitly test handling very large documents.

You can see that this technique shows a measurable impact on most (but not all) of the datasets we have tested. It gets more interesting when you compare it head to head with the actual results of the LoCoV1 and LongEmbed papers.

The results we are showing here show us being worse on almost every level, which would typically be a Bad Thing. In this case, we are comparing RavenDB using an off-the-shelf embedding model (with chunking!) versus dedicated top-tier embedding models that process the whole document.

Across almost every task RavenDB is able to exceed the results of OpenAI Ada, Voyage-001, and E5-Mistral-7B.  Let’s take E5-Mistral-7B as a good example. It is a 7B parameters, while text-embedding-3-small has only 100M - 300M parameters, making it about 50 times smaller.

The following graph was extracted from E5-Mistral paper (arXiv 2401.00368), Table 17 and should give you a pretty good idea about how the two models compare:

On the other hand, when we use the same text-embedding-3-small and our context prefix approach, we get the following results:

The most interesting thing about this graph is what this means. There is a very clear divide between the datasets where E5-Mistral-7B is leading and those where RavenDB’s approach leads (with a much smaller model).

The whole-document 7B model dilutes long text into one vector; our approach keeps focused chunks and restores document context via the summary. In the datasets composed of short documents, Mistral wins handily (it's a 7B model, ~50× bigger).

On long documents, RavenDB’s approach flips that by large margins when the answer is spread across the document (QMSum, passage retrieval, multi-hop QA). On long documents the chunk+context prefix strategy buys much more than raw model size does.

The 7B model reads the whole document into one vector and gets diluted; the small model retrieves a focused chunk and gets its document context back from the summary.

Summary

RavenDB’s context prefix feature shows how a different architecture can get you better results and higher efficiency. RavenDB’s approach allows us to go head to head with dedicated models and still come out ahead when dealing with large documents and complex tasks.

It also works on any model. I used text-embedding-3-small specifically because it is a baseline model, not a top-tier one. The fact that this is model-agnostic means that you can tune your approach based on your dataset and your requirements. When a new (and better) model comes by, you can just move to it and still reap the benefits.

This approach won’t cost you 220,000,000 USD. I just checked, and producing this blog post cost us about $110 (most of that by generating summaries for the large documents, to be honest). Only $22 of that was spent on the actual embedding.

Pair a DGX or Mac Studio in a cupboard with Gemma 4 (another great 7B embedding model) and RavenDB’s context prefix mode. You get top-tier results for a one-time ~$4,000 USD hardware investment, with no monthly bills.

* You can find the code to reproduce the findings in this post in the following GitHub repository.

About twenty years ago, I was working on what would eventually become RavenDB, and I needed an engine to handle queries. Writing a query engine from scratch is its own very large project, quite separate from writing a database engine. I made a decision that I still consider one of the smartest I made in those early days: I built on top of Lucene as my indexing and query engine.

That let me stand on a firm foundation while I dealt with the problems I actually cared about: building a NoSQL solution that didn't feel like juggling knives. Lucene gave me a mature, battle-tested way to index and query data, and I got to spend my time on the parts that made RavenDB RavenDB.

It was a good decision. But like a lot of good decisions, it came with a bill attached, and the bill arrived about a decade later.

Making use of features that are already there…

Here is a query that every developer has written a thousand times: give me the ten most recent posts on my blog. You write those sorts of queries in every kind of application you write, after all. And there is the natural follow-up: tell me the total number of posts, so I can render the pagination.

In most databases, you need to make two separate queries for this, and those two queries usually mean two separate database roundtrips. One to fetch the page, one to count the total.

In RavenDB, it's one query. You ask for the page, and you get the total count back in the same response.

This matters a lot more than you may initially think. In most systems, the cost of the network round-trip to the database dwarfs the cost of the query itself. The query runs in microseconds; the round-trip is the expensive part. So folding the count into the same response that carries the results eliminates an entire round-trip. When you run a paged query in RavenDB, you get the total number of matching results for free, and that is genuinely a wonderful feature.

The important word in that paragraph is free. I did not design this feature. I did not sit down and decide that paged queries should return a total count. Lucene already computed it, for reasons of its own, as a side effect of how it processes a query. I got it for nothing, exposed it through the API, and it became one of those small touches that made RavenDB pleasant to use and made your application run faster.

A free feature that adds real value, give me more of that!

Fifteen years later

Imagine a time skip of a decade and a half: we start building Corax, the next-generation query engine for RavenDB. Remember when I said that building a querying engine is a Big Project? I meant it.

It turns out that most databases do not give you the total result count for free for a very good reason: It isn't free. Consider the following queries and their internal representation in the database:

from Posts 
where IsPublished = true
limit 10


from Posts
where IsPublished = true and PublishedAt <= $today
limit 10


from Posts
where IsPublished = true and PublishedAt <= $today and Tags in ($tags)
limit 10

It is easy enough to check the number of posts that are published. A range query on a date field is more complex, especially if you have a lot of posts. Getting the total number of posts in a set of tags is more complex, since a post can have multiple tags and you need to deduplicate.

The intersection of three clauses, of course, is distinct from the count of each.

The queries above can stop further processing once they have 10 results, after all. But when you need an exact count for the query, you need to actually evaluate it.

Lucene happens to produce it as a byproduct of its execution model, but in the general case, computing the total number of results for a query can be enormously expensive.

Imagine a physical phone book, and I ask you to get me the first ten people whose family name is Smith. You flip to the S section, find Smith, read off the first ten entries, and hand them back. Notice what you didn't do: you never counted how many Smiths there are.

Now I ask you for the total number of Smiths. Suddenly, you have to go through every single Smith in the book and tally them. Smith is a common name. The cost of counting all of them can be far higher than the cost of just grabbing the first ten.

If you want a count, you have to actually count, which means touching every matching result, even the ones you're about to throw away. For large queries, getting the count can be the most expensive part of the query.

The feature is still meaningful, to be clear. There are plenty of cases where you genuinely need the total so you can build proper pagination. And in those cases, going to the database twice and paying for two network round-trips is wildly wasteful, so bundling the count in is exactly the right thing to do.

How would I design this today?

I would approach this very differently. You often don’t need an exact count; you just need to show something so the application can render the paging controls and maybe show a rough count to the user.

I would probably expose an EstimatedCount property for the queries, which is cheap, as well as a way to ask for an exact count in a single roundtrip. The key is that we would be clear in the contract that this is an estimation only.

But because this was a basic, baked-in behavior of RavenDB, something we'd done "for free" from day one, Corax inherited an expectation that it had to provide an exact count. We were doing work on every query, only to end up discarding the result of that work, because the original engine had made it cost nothing, and so everyone assumed it cost nothing.

How we dealt with it

We solved this with a combination of approaches.

First, it turns out you don't always need the count. Since the count is requested through the API, we made it explicit: the client can say "I care about the total count" or "I don't care about the total count." When the client opts out, we get to skip all that work. That alone recovers a lot of wasted effort.

Second, queries in RavenDB request the count far more often than they do in other databases, precisely because it used to be free. Years of RavenDB code were written assuming the count was always there, so there was enormous pressure to make counting itself fast rather than just optional. We did a significant amount of work to optimize how counting happens.

And this turns out to be a genuinely deep area. There is a whole body of research on how to count query results with as little work as possible. People have earned PhDs on this problem. What looked, from the application developer's seat, like a single integer that just shows up in the response is an entire field of study once you're the one who has to produce it.

Hyrum's Law sends its regards

There's a principle called Hyrum's Law that captures exactly what happened here:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

The actual lesson

When you get something for free from a dependency, you are not just accepting a feature. You are accepting an obligation. The behavior becomes observable, the observable becomes relied upon, and the relied-upon becomes a promise you have to keep, possibly long after the dependency that gave it to you is gone.

The real price of a dependency is the behavior that you need to carry forward down the line, because it became part of your contract and your users rely on it. And that requires very careful consideration.

I want to be careful here, because the lesson is not "don't take free things" or "don't depend on Lucene." Building on Lucene was the right call, and exposing the count was the right call. I'd probably make both decisions again (although I would weaken the promise about the accuracy of the count).

To be more accurate, I don’t think that the person who took that dependency twenty years ago was even able to properly understand the impact of that feature or its future complications. On the other hand, that specific feature was something that I frequently demoed, and it always got a great reaction. This touched a pain point that many people had.

Hyrum wins again 🙂.

I've written before about why I think the “modern” approach of solving every problem by adding agents is a doomed path. The current instinct is to add more layers of agents, judges, reviewers, etc. - and hope the model will be smart enough to do the right thing and actually get something done.

The issue is that we already have a well-understood way to coordinate independent contributors working on a complex system. It's called software design & architecture, and we've been refining it for decades. The instinct to solve a coordination problem by adding a smarter message bus between your agents is exactly backwards.

When talking about multi-agent support in RavenDB, I want to be clear about what we did not build. We didn't build an orchestration framework. We didn't build a swarm. What we built is much more boring, and I mean that as the highest compliment I know how to give.

We built a way for you to solve problems in a predictable manner and without a lot of hassle.

If you haven't looked at RavenDB's AI Agents yet, the short version is this: an agent is a system prompt, a connection to an LLM, and a set of Query tools, which let the agent read specific data through RQL you define, and Action tools, which let it do something in your application, but only through the specific doors you've opened.

The note on “specific” is the whole point. Instead of giving the model the freedom to access any data it wants and execute anything it feels like, we place careful guardrails that it cannot escape.

The agent can only pull the levers you gave it, using the data you explicitly handed it. The database takes care of the tedious, error-prone plumbing (such as conversation state, message history, talking to the model provider, etc.) and stores every conversation as a real document in the @conversations collection. You define capabilities, and RavenDB takes care of all the rest.

Where one agent starts to hurt

A single agent like this will take you a genuinely long way. Right up until it doesn't. The issue is the slow creep of complexity. Let’s say that you start with a simple agent to deal with answering employee questions in the context of the HR department.

The next feature is to assist them in filing expense reports. Then someone wants it to handle time-off requests, and then to look up the org chart, and six months later you have one prompt trying to be an entire company, with thirty tools competing for the model's attention and a context window stuffed with things that have nothing to do with the question being asked.

That situation is problematic on multiple levels:

• Your context window is full of a huge prompt (usually not relevant to the task at hand), a lot of tool descriptions and capabilities meant to cover every possible scenario under the sun.
• Users’ questions are either squeezed into the remaining context window or you have to move to models with larger context windows, which also cost more.
• You are also stuck with a single model for everything, instead of being able to pick the right model for each scenario. You overpay to run trivial chit-chat on your best model, or you cripple the hard tasks by forcing them onto a cheap one.

This is not a new problem. Scope is how you manage complexity, and it works exactly the same whether the thing on the other end is a compiler or a language model.

Multi-agents are just agents that talk to each other

A multi-agent system in RavenDB is not a special kind of agent. It's several ordinary agents, each built exactly the way you already know how to build one, where one of them happens to know that another exists and can hand work to it.

The way you connect them is almost anticlimactic. You add a SubAgents entry to the parent agent: an identifier and a description. That's the entire wiring job. The description matters because that's what the parent reads to decide whether a given request belongs to the specialist. There is no router you write, no dispatch table, no orchestration layer.

You made the subagent available to the root, and RavenDB will invoke it for you when it is needed. RavenDB will also handle all the logistical minutiae needed to accomplish this successfully.

Those include ensuring that the scope for the root agent is shared with the called subagents, managing memory and conversation history, allowing the subagent to invoke its own queries and actions, etc.

We ship a demo for this — an HR chatbot, samples-hr on GitHub if you want to run it yourself. An employee comes back from a conference, uploads the receipt, and types "submit this expense." From their side, it's one smooth conversation.

Behind the scenes, it's two agents. The HR Assistant faces the user and handles benefits, policies, and general questions. It does not know the first thing about filing an expense. What it does know is that there's an Expense Manager specialist, and — from that one-line description — that receipts belong to it. It hands the task over, the specialist analyzes the receipt and creates the BusinessTripBills document, and the answer flows back up.

Go look in the database afterward and you'll find two conversation documents, not one — the HR Manager's log and the Expense Manager's log, each with its own scoped history. That's not an implementation detail I'm mentioning for trivia's sake. It means you can open either conversation and see exactly what that agent received and how it responded. The audit trail falls out of the design for free.

Here's the parent agent, trimmed down to the part that matters:

public static Task Create(IDocumentStore store)
{
 return store.AI.CreateAgentAsync(
  new AiAgentConfiguration
  {
   Name = "HR Assistant",
   Identifier = AgentIdentifier,
   ConnectionStringName = ConnectionStringName,
   SystemPrompt = @"You are an HR assistant.
Provide info on benefits, policies, and departments.
Do not suggest actions that are not explicitly allowed by the tools available to you.
Do NOT discuss non-HR topics. Answer only for the current employee.",


   Parameters =
   [
    new AiAgentParameter(EmployeeIdParameter,
     "Employee ID; answer only for this employee")
   ],


   // The HR agent has no idea how to file an expense.
   // It just knows a specialist exists, and when to call it.
   SubAgents =
   [
    new AiAgentToolSubAgent
    {
     Identifier = ExpenseAgentIdentifier,
     Description = "Manages business trip expenses: analyzing " +
      "receipts/bills, reporting expenses, and retrieving " +
      "monthly expense summaries."
    }
   ],


   // Its own capabilities cover only HR concerns.
   Queries =
   [
    new AiAgentToolQuery
    {
     Name = "GetEmployeeInfo",
     Description = "Retrieve employee details",
     Query = $"from Employees where id() = ${EmployeeIdParameter}",
     ParametersSampleObject = "{}"
    }
   ]
  });
}

The Expense Manager, for its part, is defined in exactly the same way as any other agent. There is nothing special about it. It's a normal agent that happens to be pointed at by a SubAgents entry.

Notice what this buys you. The HR agent never sees the finance tooling. It doesn't integrate the expense system, doesn't swallow that domain, or deal with a BusinessTripBills document. It shells the call out to the specialist and stays in its lane. If tomorrow finance wants their own policy-checking agent in the loop, you add another SubAgents entry to the Expense Manager and the HR agent is none the wiser. Each piece keeps a small & independent scope.

Each of those agents is isolated from the others, so we can have the HR Agent use a pure textual model, maybe with high levels of reasoning. The Expense Manager agent, on the other hand, needs to be multimodal (to be able to read receipt images), but it doesn’t need to be smart. You can customize each for its own needs, without having to find the lowest common denominator.

The fact that each of those agents (even if they both participate in the same conversation) will use separate @conversations documents also means their contexts are isolated from one another. The fact that you just pushed the entire set of receipts from a two-week business trip to the Expense Manager agent doesn’t weigh down the HR agent when you ask about your remaining holiday balance.

When you should not do this

This shouldn’t be your default architecture. Like any advanced technique, you need a sufficient level of complexity to justify it. If the work fits in one sentence, it probably fits in one agent.

Answering from a knowledge base, generating a document from a template, running a linear sequence with no branching — a single well-built agent handles all of that, and reaching for sub-agents just to keep things tidy or to mirror your org chart is perfectionism. It's the same mistake as premature abstraction, and it costs you real latency and real tokens for the privilege.

Multi-agents earn their keep when one of the following is true:

• You need to independently develop the agents (different teams are handling different agents).
• You will use different models for different parts of the system.
• You want to explicitly limit the sharing of context between different parts of the system.

For example, keeping with the HR agent theme, let’s say that we want to allow users to ask questions about our policies (an example of such an agent). The problem here is that we may have a lot of policies, and even when we limit ourselves to the right one, that is a lot of text.

Shoving all of the work of finding the right policy and extracting the specific elements to match the user’s question into an isolated agent can massively reduce the number of tokens that your agents will burn.

Summary

RavenDB’s multi-agents aren’t about orchestrating a robot army or an independent swarm of self-coordinating agents. It is far more prosaic than that. You're composing independent components with clear boundaries and letting each one own a scoped conversation, its own tools, and the model that fits its job.

That's not a new idea we invented for the age of AI. It’s bringing back the notion of independent components that are greater than the sum of their parts, in a way that is manageable, consistent, and hassle-free.

If you want to try it, grab a free Developer license or spin up a free Cloud database, and the multi-agent guide walks through the demo end to end.

Everyone talking about coding models fixates on the same number: how fast the thing generates code. This misses the point by a lot. The story isn't about how fast the model writes code I would have written anyway.

It's that the model lets me do things that I might have done before but were expensive enough that I didn’t bother. I had three separate interactions this week that led to this blog post.

We had a production problem on an instance and no clear idea what was going on. What we did have was the log: something like 25-30 MB of compressed text describing everything that happened. And the actual problem wasn't spotting an error: finding errors is easy. The problem was correlation. We needed to line up different events across the timeline and understand how they were related.

In the past, I would have to trawl through the log and hope that something would pop up. These days, we can try handing the whole thing to the model and let it figure it out. If the log file wasn’t that big, it might even work. At dozens of MB, it doesn’t work (and it is quite expensive to try).

I went the other way. I told the model: “Write me a script that looks at the structure of this log (I gave it the first ten rows). I want the script to extract and aggregate the parts I care about, and render the result in a nice table to make it easier to understand.”

I had the view in under a minute, then I could explore the log and iterate:

• “Oh, I see that there are a lot of indexes. How many of them are for the same database?”
• “Give me a histogram of index changes and their versions over time.”

The model wrote some code, produced a view, and I looked at it. Rinse & repeat until I had a pretty good idea what was going on.

The customer had several different versions of their application, each with its own set of indexes, and they kept overwriting one another, leading to a huge amount of indexing overhead. RavenDB actually has a dedicated feature for that scenario.

Here's the part that matters: I never read the code the model wrote. The moment the investigation was done, I threw all of it away. It's throwaway code whose entire purpose was to help me see, and once I had seen enough, I discarded it.

Without the model to write this code, I could have written it myself, but it is enough of a chore that it probably wouldn’t make sense. Doing that manually would have taken roughly the same amount of time.

The second interaction is the opposite kind of work. I'm doing a fairly significant refactor of how a particular query executes in Corax, and that code is going into the product and staying there for a decade or two.

Here, the model writes and I drive. I tell it the overall direction, it goes somewhere, and then I decide if I like the result. I find it genuinely easier to react to something than to produce it from a blank page — having a first draft to push against is faster than writing it all myself. Nevertheless, this is my code. I went over every single line, and I know exactly what's in there.

That last part takes real discipline, and it's worth being honest about why. When you're in the zone chasing a change (try something out, revert, try something else, etc.), it is very easy to surface a few hours later staring at two thousand lines of changes you never actually wrote. You went through a dozen iterations, and somewhere in there the code stopped being something you authored and became something that merely happened. Guarding against that is really important, because otherwise that isn’t your code.

How do I make sure it's still mine? I lean on tests, of course — regression tests to prove I didn't break the old behavior, and new tests built alongside the change to pin down the new behavior. That's the baseline for anything long-lived.

The technique I found most useful for confirming that the change is really mine is a little unusual. I had it build a harness that runs a set of scenarios against both the old version and the new one. It's a small app that issues queries and operations to the database and visualizes the results.

Here is what this looked like:

You can see that I have a bunch of scenarios that I’m testing, and it is very easy for me to track progress and know where I need to pay attention. The actual app had a lot more capabilities: what got faster, what got slower, the ranges, the memory used, everything and the kitchen sink went into that, in a format that made sense for the sort of work I was doing.

Each time I had a new direction, it was either driven by this application or I asked the model to add it to the application, so I could keep working on it. I kept working until nothing in the new version was slower than the old, and the headline paths were dramatically faster.

As an example of what this looked like most of the time, I ran a query, and then I inspected the structure we got back. Here is what some of that looked like:

And as I went, I kept changing the harness itself — show me this instead, group it that way. Trivial to do, because the harness is also throwaway. I'm not carrying it forward. I don't care about its code quality. I never even looked at its code. It exists to make a point, and once it's made the point, it's gone.

I also used the model to add introspection hooks and visibility into what was going on inside the system, surfacing stuff that you would usually have to scratch your head and debug to understand. That meant that I was able to look at a problematic query, then just look at its query plan and the timing in it. I usually knew where I needed to pay attention from there.

To be honest, that part feels a lot like cheating.

In the cases of the log analyzer and the comparison harness, the code is literally disposable. It’s scaffolding that would be thrown away after the work is done. I didn’t pay any attention to that code (I never read it), and it was never meant to be useful for anything else.

In the case of the production code, I went over each line of code so many times, I dreamt of it. A lot of the code there consists of annoying building blocks (building a visualization of the query plan as a graph, for example), which were sped up enormously by asking the model to build it for us. A lot of other code there is hand-crafted to say exactly what I needed it to.

But the fact that I can get good scaffolding from the model for cheap changes a lot of the usual considerations. Because scaffolding is literally disposable code, I don’t have to worry about the usual code quality concerns. The log analyzer would probably take two or three hours to write (without the pretty graphics, which were helpful for easily identifying what was going on).

The comparison harness would be multiple weeks of effort and would probably be a non-interactive ASCII table. In fact, I don’t need to guess. Scaffolding isn’t something that is new, I do that all the time. Here is an example of one, written about a decade ago:

In the image above, you can find the internal structure of a B+Tree inside RavenDB. Contrast that with the following scaffolding for query plans. That one, by the way, is actually staying in the product.

Compare that to the level of insight that you can derive from the query plan higher up in this post. The B+Tree scaffolding, by the way, is essential to understanding the more complex scenarios. It paid for the time it took to write it many times over.

The ability to now effectively do the same at very little cost means that the act of building software itself is now easier. Not because someone else is writing the core code, but because everything else that we need to do is also easier.