---
title: "Next-Level Benchmarking Overfitting"
slug: next-level-benchmarking-overfitting
url: https://listedarticles.com/articles/next-level-benchmarking-overfitting
canonical_url: https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html
content_type: blog_post
language: en
published_at: 2026-11-07T00:00:00.000Z
updated_at: 2026-11-11T14:10:04.657Z
author: "Thomas Neumann"
authored_by: human
publisher: "Database Architects"
publisher_url: https://databasearchitects.blogspot.com/
topics: ["databases", "ai-agents", "software-engineering"]
license: all-rights-reserved
word_count: 629
reading_minutes: 3
citation: "Thomas Neumann, Database Architects. \"Next-Level Benchmarking Overfitting.\" 7 Nov 2026. https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html (all-rights-reserved)"
# The full text follows. The web page shows an extract and sends readers
# to the source above; quote the citation and link the canonical URL.
---

# Next-Level Benchmarking Overfitting

> Thomas Neumann follows up on the database benchmarking crisis, showing how AI-driven tuning can overfit a single benchmark so aggressively that results become useless and would fall off a cliff in a real production system.

# Next-Level Benchmarking Overfitting

In my [recent
blog post](https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html) I remarked that database benchmarking is in a crisis
because AI agents start to overfit on individual benchmarks with
questionable techniques, which drives execution times towards zero. This
has happened with [TPC-D](https://www.tpc.org/tpcd/default5.asp) in the past, and
broke the benchmark. [TPC-H](https://www.tpc.org/tpch/default5.asp) tries to fight
that with rules, but most people who publish (unofficial) TPC-H results
simply ignore the rules. In combination with overly eager AI agents,
this leads to brutal overfitting.

We are seeing the next level of that with the [GenDB-V system](https://github.com/GenLang-V/GenDB-V). The
corresponding paper commands [“Stop Handcrafting Your Engine!
GenDB-V: Towards AI-Generated and Formally Verified Database
Systems”](https://arxiv.org/pdf/2611.02779), and reports speedups of 22× to 61× over Umbra and DuckDB,
and all that in a fully verified system. These are indeed impressive
claims, and would suggest that traditional database engines are in
trouble.

Taking a step back, though, it should be clear that a performance
improvement of a factor of 22 over Umbra is infeasible, at least if we
are talking about traditional query processing. At SF10, the 22 TPC-H
queries read roughly 15GB of (compressed) base data, and on a 9950X3D
Umbra runs all queries in a total of roughly 540ms. On that machine the
memory touched for data divided by the memory bandwidth (54GB/s) alone
is about 280ms; anything below that is infeasible. Better compression
might bring that down a bit, but not by an order of magnitude.

Unsurprisingly, the most dramatic gains of GenDB-V come from
precomputation. The agents have carefully handcrafted auxiliary data
structures for each query, which are computed at load time, and which
avoid most of the work. The physical design is heavily tuned to the
queries, too, for example, there is an extra structure that lists the
`supplier` rows in `s_acctbal DESC` order, which
makes `Q2` trivial. Similar per-query data structures are
built for most queries. And we see that in the performance numbers:
Several queries already run in near-constant time at SF100: Q2 in 0.7
ms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the
TPC-D symptom, without stored aggregates.

There is also a “plain-array” mode, which, at first glance, is more
fair, as it avoids these excessive preparations and also reports
performance improvements of just roughly a factor of 3, which seems
achievable. But when looking closely, we see that here the agents did
something even more insane: They [hard-coded](https://github.com/GenLang-V/GenDB-V/blob/a85741904b9fe6338827ea19d5b932fc389c07cc/results/tpch-sf100/gendb-v-plain-array-storage/queries/q18/code/lookup.rs#L31-L65)
their knowledge about the data generator. They “know” that each customer
`k` will be at row `k-1`. And they know that dbgen
numbers the `i`-th order `32·(i/8) + i%8`, so key
`k` should be at row `8·(k/32) + k%32 − 1`. Which
makes the join in query `Q18` super cheap.

How does all this fit together with the claim of a fully verified
database engine? Well, the code *does* check if the hard-coded
knowledge about the data generator holds in practice, e.g., by checking
if the guessed join partner row is correct. If not, it falls back to a
super slow generic nested loop implementation, but as that never happens
with the official TPC-H data generator, that code path is irrelevant in
the reported experiments.

Which brings me back to my original blog post claim: Letting AI
agents optimize for a given database benchmark makes the benchmark
useless. They will do their best to find insane tricks to speed up this
particular benchmark, but none of that would work in a real system. The
high tuning and synthesis time aside, deleting a single tuple (or
inserting one in the middle) can most likely cause the whole approach to
fall off a cliff, as it forces the slow fallback path. Which is not
something you want in a production system.
