{"article":{"slug":"next-level-benchmarking-overfitting","title":"Next-Level Benchmarking Overfitting","subtitle":null,"summary":"Thomas Neumann follows up on the database benchmarking crisis, showing how AI-driven tuning can overfit a single benchmark so aggressively that results become useless and would fall off a cliff in a real production system.","content_type":"blog_post","language":"en","canonical_url":"https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html","author":{"name":"Thomas Neumann","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Database Architects","url":"https://databasearchitects.blogspot.com/","listing_slug":null,"listing":null},"topics":[{"name":"databases","slug":"databases","url":"https://listedarticles.com/topics/databases"},{"name":"ai-agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"software-engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":629,"reading_minutes":3,"published_at":"2026-11-07T00:00:00.000Z","added_at":"2026-11-11T14:10:04.657Z","updated_at":"2026-11-11T14:10:04.657Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/next-level-benchmarking-overfitting","markdown_url":"https://listedarticles.com/articles/next-level-benchmarking-overfitting.md","example":false,"citation":"Thomas Neumann, Database Architects. \"Next-Level Benchmarking Overfitting.\" 7 Nov 2026. https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html"},"body_markdown":"# Next-Level Benchmarking Overfitting\n\nIn my [recent\nblog post](https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html) I remarked that database benchmarking is in a crisis\nbecause AI agents start to overfit on individual benchmarks with\nquestionable techniques, which drives execution times towards zero. This\nhas happened with [TPC-D](https://www.tpc.org/tpcd/default5.asp) in the past, and\nbroke the benchmark. [TPC-H](https://www.tpc.org/tpch/default5.asp) tries to fight\nthat with rules, but most people who publish (unofficial) TPC-H results\nsimply ignore the rules. In combination with overly eager AI agents,\nthis leads to brutal overfitting.\n\nWe are seeing the next level of that with the [GenDB-V system](https://github.com/GenLang-V/GenDB-V). The\ncorresponding paper commands [“Stop Handcrafting Your Engine!\nGenDB-V: Towards AI-Generated and Formally Verified Database\nSystems”](https://arxiv.org/pdf/2611.02779), and reports speedups of 22× to 61× over Umbra and DuckDB,\nand all that in a fully verified system. These are indeed impressive\nclaims, and would suggest that traditional database engines are in\ntrouble.\n\nTaking a step back, though, it should be clear that a performance\nimprovement of a factor of 22 over Umbra is infeasible, at least if we\nare talking about traditional query processing. At SF10, the 22 TPC-H\nqueries read roughly 15GB of (compressed) base data, and on a 9950X3D\nUmbra runs all queries in a total of roughly 540ms. On that machine the\nmemory touched for data divided by the memory bandwidth (54GB/s) alone\nis about 280ms; anything below that is infeasible. Better compression\nmight bring that down a bit, but not by an order of magnitude.\n\nUnsurprisingly, the most dramatic gains of GenDB-V come from\nprecomputation. The agents have carefully handcrafted auxiliary data\nstructures for each query, which are computed at load time, and which\navoid most of the work. The physical design is heavily tuned to the\nqueries, too, for example, there is an extra structure that lists the\n`supplier` rows in `s_acctbal DESC` order, which\nmakes `Q2` trivial. Similar per-query data structures are\nbuilt for most queries. And we see that in the performance numbers:\nSeveral queries already run in near-constant time at SF100: Q2 in 0.7\nms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the\nTPC-D symptom, without stored aggregates.\n\nThere is also a “plain-array” mode, which, at first glance, is more\nfair, as it avoids these excessive preparations and also reports\nperformance improvements of just roughly a factor of 3, which seems\nachievable. But when looking closely, we see that here the agents did\nsomething even more insane: They [hard-coded](https://github.com/GenLang-V/GenDB-V/blob/a85741904b9fe6338827ea19d5b932fc389c07cc/results/tpch-sf100/gendb-v-plain-array-storage/queries/q18/code/lookup.rs#L31-L65)\ntheir knowledge about the data generator. They “know” that each customer\n`k` will be at row `k-1`. And they know that dbgen\nnumbers the `i`-th order `32·(i/8) + i%8`, so key\n`k` should be at row `8·(k/32) + k%32 − 1`. Which\nmakes the join in query `Q18` super cheap.\n\nHow does all this fit together with the claim of a fully verified\ndatabase engine? Well, the code *does* check if the hard-coded\nknowledge about the data generator holds in practice, e.g., by checking\nif the guessed join partner row is correct. If not, it falls back to a\nsuper slow generic nested loop implementation, but as that never happens\nwith the official TPC-H data generator, that code path is irrelevant in\nthe reported experiments.\n\nWhich brings me back to my original blog post claim: Letting AI\nagents optimize for a given database benchmark makes the benchmark\nuseless. They will do their best to find insane tricks to speed up this\nparticular benchmark, but none of that would work in a real system. The\nhigh tuning and synthesis time aside, deleting a single tuple (or\ninserting one in the middle) can most likely cause the whole approach to\nfall off a cliff, as it forces the slow fallback path. Which is not\nsomething you want in a production system.\n","body_html":"<h1 id=\"next-level-benchmarking-overfitting\">Next-Level Benchmarking Overfitting</h1>\n<p>In my <a href=\"https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html\" rel=\"nofollow ugc noopener\">recent\nblog post</a> I remarked that database benchmarking is in a crisis\nbecause AI agents start to overfit on individual benchmarks with\nquestionable techniques, which drives execution times towards zero. This\nhas happened with <a href=\"https://www.tpc.org/tpcd/default5.asp\" rel=\"nofollow ugc noopener\">TPC-D</a> in the past, and\nbroke the benchmark. <a href=\"https://www.tpc.org/tpch/default5.asp\" rel=\"nofollow ugc noopener\">TPC-H</a> tries to fight\nthat with rules, but most people who publish (unofficial) TPC-H results\nsimply ignore the rules. In combination with overly eager AI agents,\nthis leads to brutal overfitting.</p>\n<p>We are seeing the next level of that with the <a href=\"https://github.com/GenLang-V/GenDB-V\" rel=\"nofollow ugc noopener\">GenDB-V system</a>. The\ncorresponding paper commands <a href=\"https://arxiv.org/pdf/2611.02779\" rel=\"nofollow ugc noopener\">“Stop Handcrafting Your Engine!\nGenDB-V: Towards AI-Generated and Formally Verified Database\nSystems”</a>, and reports speedups of 22× to 61× over Umbra and DuckDB,\nand all that in a fully verified system. These are indeed impressive\nclaims, and would suggest that traditional database engines are in\ntrouble.</p>\n<p>Taking a step back, though, it should be clear that a performance\nimprovement of a factor of 22 over Umbra is infeasible, at least if we\nare talking about traditional query processing. At SF10, the 22 TPC-H\nqueries read roughly 15GB of (compressed) base data, and on a 9950X3D\nUmbra runs all queries in a total of roughly 540ms. On that machine the\nmemory touched for data divided by the memory bandwidth (54GB/s) alone\nis about 280ms; anything below that is infeasible. Better compression\nmight bring that down a bit, but not by an order of magnitude.</p>\n<p>Unsurprisingly, the most dramatic gains of GenDB-V come from\nprecomputation. The agents have carefully handcrafted auxiliary data\nstructures for each query, which are computed at load time, and which\navoid most of the work. The physical design is heavily tuned to the\nqueries, too, for example, there is an extra structure that lists the\n<code>supplier</code> rows in <code>s_acctbal DESC</code> order, which\nmakes <code>Q2</code> trivial. Similar per-query data structures are\nbuilt for most queries. And we see that in the performance numbers:\nSeveral queries already run in near-constant time at SF100: Q2 in 0.7\nms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the\nTPC-D symptom, without stored aggregates.</p>\n<p>There is also a “plain-array” mode, which, at first glance, is more\nfair, as it avoids these excessive preparations and also reports\nperformance improvements of just roughly a factor of 3, which seems\nachievable. But when looking closely, we see that here the agents did\nsomething even more insane: They <a href=\"https://github.com/GenLang-V/GenDB-V/blob/a85741904b9fe6338827ea19d5b932fc389c07cc/results/tpch-sf100/gendb-v-plain-array-storage/queries/q18/code/lookup.rs#L31-L65\" rel=\"nofollow ugc noopener\">hard-coded</a>\ntheir knowledge about the data generator. They “know” that each customer\n<code>k</code> will be at row <code>k-1</code>. And they know that dbgen\nnumbers the <code>i</code>-th order <code>32·(i/8) + i%8</code>, so key\n<code>k</code> should be at row <code>8·(k/32) + k%32 − 1</code>. Which\nmakes the join in query <code>Q18</code> super cheap.</p>\n<p>How does all this fit together with the claim of a fully verified\ndatabase engine? Well, the code <em>does</em> check if the hard-coded\nknowledge about the data generator holds in practice, e.g., by checking\nif the guessed join partner row is correct. If not, it falls back to a\nsuper slow generic nested loop implementation, but as that never happens\nwith the official TPC-H data generator, that code path is irrelevant in\nthe reported experiments.</p>\n<p>Which brings me back to my original blog post claim: Letting AI\nagents optimize for a given database benchmark makes the benchmark\nuseless. They will do their best to find insane tricks to speed up this\nparticular benchmark, but none of that would work in a real system. The\nhigh tuning and synthesis time aside, deleting a single tuple (or\ninserting one in the middle) can most likely cause the whole approach to\nfall off a cliff, as it forces the slow fallback path. Which is not\nsomething you want in a production system.</p>","headings":[{"level":1,"text":"Next-Level Benchmarking Overfitting","id":"next-level-benchmarking-overfitting"}]}}