The shelf life of LLM findings

Owen Queen, James Zou

Stanford University

November 2, 2026

How long does a published finding about LLMs remain true? We revisited 50 widely cited findings about LLMs across 20 models. Within 15 months of publication, most of the LLM claims we tested either no longer held or showed substantially diminished effects. What we know about AI may have a surprisingly short shelf life.

Fitted effects on September 30, 2026. How we classify claims.

In 2023, Walters and Wilder asked ChatGPT to write short literature reviews and then tried to find every work it cited. At the time, ChatGPT used GPT-3.5, and 55% of its citations were fake (C11). We sent the same prompts to recent models (GPT-6.1-Sol and Claude Sonnet 5.5) and found that they rarely output fabricated citations, though some models still do. We cannot assume claims made on yesterday’s LLMs hold on today’s more capable models.