There is no posttraining data commons

October 15, 2026 · David Hall ·

EXPLAINER

We got extraordinarily lucky with pretraining data. Training large language models required a truly vast corpus of text, and it was just kind of lying around.

Not exactly by accident. For the last 50 or so years, humanity has been busily creating and digitizing an enormous public record of human knowledge. Most obviously, Wikipedia’s editors have assembled the most comprehensive encyclopedia humanity has ever produced. A vigorous open source movement created a couple petabytes of software source code. Communities on Stack Exchange spent more than a decade asking questions, arguing about answers, and carefully curating the results. Scientists uploaded their papers to ArXiv. Project Gutenberg—almost as old as the internet itself—produced digital editions of out-of-copyright books.