Thomas Neumann · Database Architects · October 27, 2026
\nGoodhart’s\nlaw, that any measurable target stops being useful when people start\noptimizing for it, very much holds in database benchmarking. That is not\nnew, of course. Famously, TPC-D got retired when\npeople optimized so much for it that the query runtimes converged\ntowards zero due to excessive caching. TPC-H fixed that by\ndeclaring itself to be an ad-hoc benchmark, and explicitly forbidding\nthings like exploiting knowledge about the workload or caching results\n(and a number of other optimizations).
\nWe now see the same problem with the ClickBench benchmark,\nheavily accelerated by today’s use of AI code generation. If you tell an\nAI to make the queries faster, it will make the queries faster,\nregardless of whether this is a generally plausible approach or not. For\nexample, the pgrust implementation performs group-by aggregations on\nstring columns purely on the hash value, ignoring the long string\ncontent. That is a great performance boost, of course, and works for\nthis benchmark, but can produce wrong results. Is it faster? Yes. Is it\na plausible optimization? Probably not. And it does many other things:\nFor example, ClickBench repeats certain predicates across queries, and\nthus pgrust caches the predicate, completely eliminating the query cost\nthe second time. Or for query Q17, which returns tuples grouped by\nuserid, it makes use of the fact that a) tuples are roughly clustered by\nuserid, and b) the query specifies a limit clause without order by. Thus\nany userid will do, and it never reads most of the data. A human would\nnot do these tiny micro-optimizations, because they are unlikely to be\nrelevant in general use cases, but an AI can of course look at each\nquery in detail and tune for it.
\nPart of the problem is, of course, ClickBench itself. The benchmark\nis not well specified in what is permitted and what is not, there is no\nupdate stream, the queries are very simple, share predicates, are\nunderspecified, etc. But the fundamental problem is independent of that\nconcrete benchmark: The moment we start aggressively optimizing for a\nspecific benchmark, the benchmark stops being very useful. Historically,\nthat pressure was reduced a bit by the requirement that systems be\ngenerally useful. That is, they did not just want to perform well in\nbenchmarks but also to be useful for their customers. That is, of\ncourse, not a concern for vibe-coded execution engines.
\nIt is not clear what to do about it. Our SQLStorm proposal tried\nto address this by having tens of thousands of queries, which makes\noverfitting harder. No human would fine-tune their system to such a huge\nworkload. But with AI tools we could even investigate ten thousand\nqueries in detail if needed, so that is not a real defense.
\nI think that to be meaningful, a benchmark has to a) like TPC-H, be\nad-hoc, and forbid exploiting prior knowledge and any form of\nintermediate result caching. If we don’t do that, execution times\nconverge towards zero. And b) the benchmark has to include some form of\nupdate stream, as otherwise overly aggressive read-only or read-mostly\nsystems will always outperform systems that offer full transactional\nsupport. And c) the benchmark has to be truly ad-hoc, in the sense that\nthe systems tested have to be older than the benchmark. One way to fight\nGoodhart’s law would be to (automatically) come up with a new benchmark\nfrom time to time, and then measure the systems that existed before the\nbenchmark. The caching/pre-computation problem still remains, of course,\nbut at least that would prevent the over-aggressive adjustment to just\none specific benchmark.
\nInterestingly, the machine learning community has struggled with\nexactly the same problem for years, and we should probably borrow some\nof their ideas. Kaggle competitions, for example, rank submissions on a\nhidden test set that participants never see, which makes overfitting on\nthe public data pointless. Similarly, benchmarks like LiveBench regularly release new\nquestions to make sure that models cannot have seen them during\ntraining. For database benchmarking that could mean keeping the actual\nbenchmark queries secret and only publishing a representative sample for\ndevelopment, or, even better, generating a fresh workload for every\nevaluation round. SQLStorm already uses generated queries, so producing\na new query set from time to time would be fairly easy. We would\nprobably want a more generic approach (higher variety in schemas, update\nstreams, transactions, etc.), but that seems to be a direction that\nwould help keep benchmarks meaningful.
\nOriginally published at Database Architects.
","headings":[]}}