Taking a step back, though, it should be clear that a performance
improvement of a factor of 22 over Umbra is infeasible, at least if we
are talking about traditional query processing. At SF10, the 22 TPC-H
queries read roughly 15GB of (compressed) base data, and on a 9950X3D
Umbra runs all queries in a total of roughly 540ms. On that machine the
memory touched for data divided by the memory bandwidth (54GB/s) alone
is about 280ms; anything below that is infeasible. Better compression
might bring that down a bit, but not by an order of magnitude.
Unsurprisingly, the most dramatic gains of GenDB-V come from
precomputation. The agents have carefully handcrafted auxiliary data
structures for each query, which are computed at load time, and which
avoid most of the work. The physical design is heavily tuned to the
queries, too, for example, there is an extra structure that lists the
supplier rows in s_acctbal DESC order, which
makes Q2 trivial. Similar per-query data structures are
built for most queries. And we see that in the performance numbers:
Several queries already run in near-constant time at SF100: Q2 in 0.7
ms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the
TPC-D symptom, without stored aggregates.
There is also a “plain-array” mode, which, at first glance, is more
fair, as it avoids these excessive preparations and also reports
performance improvements of just roughly a factor of 3, which seems
achievable. But when looking closely, we see that here the agents did
something even more insane: They hard-coded
their knowledge about the data generator. They “know” that each customer
k will be at row k-1. And they know that dbgen
numbers the i-th order 32·(i/8) + i%8, so key
k should be at row 8·(k/32) + k%32 − 1. Which
makes the join in query Q18 super cheap.
How does all this fit together with the claim of a fully verified
database engine? Well, the code does check if the hard-coded
knowledge about the data generator holds in practice, e.g., by checking
if the guessed join partner row is correct. If not, it falls back to a
super slow generic nested loop implementation, but as that never happens
with the official TPC-H data generator, that code path is irrelevant in
the reported experiments.
Which brings me back to my original blog post claim: Letting AI
agents optimize for a given database benchmark makes the benchmark
useless. They will do their best to find insane tricks to speed up this
particular benchmark, but none of that would work in a real system. The
high tuning and synthesis time aside, deleting a single tuple (or
inserting one in the middle) can most likely cause the whole approach to
fall off a cliff, as it forces the slow fallback path. Which is not
something you want in a production system.