By Jakub Rusilko
For the last couple of weeks we have been trying to come up with a good way of assuring better stability of RavenDB. It turns out that manual tests and classical unit tests are not enough. Thus we figured that a complex test suite is needed.
We have distinguished a few separate things that we would like to be able to test. The first is the client API which can be broken into two categories:
- the .NET API
- the REST API
Within the .NET API we can further distinguish some different aspects that we would like to cover:
- sync vs async
- remote vs embeded
- self host vs IIS
Apart from that we'll also create tests for the external tools:
These will include migration tests.
In every test we'll try to cover the correctness of the results as well as the execution times. We are also planning to be able to run stress tests in single and multiple threads.
What we are currently trying to design is the way to be able to easily test different versions of RavenDB so that we can compare things across versions. Our ultimate goal is to be able to collect all the described above test data and later produce meaningful charts or graphs based on the data to help in tweaking the code. We hope that the results will not only show us whether something works or not in a given version but also how the execution time of specific operations change in different database versions.
At this point it is difficult to be more specific about the project. We are still just trying out different approaches to see what will work best for us.