Topic
Everything filed under Benchmarks, newest first.
RSS · JSON · All topics
Redis vs Valkey vs Dragonfly vs KiviDB
warefeats benchmarks Redis, Valkey, Dragonfly, and KiviDB 1.2.0 on a 64-core Graviton3, comparing throughput and RSS memory per key after a KiviDB storage rewrite.
9 min · 2,141 words
What will be the biggest tech story of 2027? what 85 AI models think
We asked 85 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 32% picked AI Agents Replace Workers. See every answer and who dissented.
12 min · 2,819 words
LLM Judges for Identity Resolution at 4x Lower Cost
We wanted to test whether LLMs and decision models, specifically Jev and Luna, could solve a hard problem that we encounter regularly (grading identity resolution results). We found that while both models performed okay out of the box (88.8–97.3% accuracy), implementing a hybrid model that combines Jev + Opus achieves frontier accuracy at > 4x lower costs. - Luna performs better than Jev out of the box, but the gap disappears on harder cases. - Jev's confidence is better calibrated. Most of its mistakes come with low confidence its right. Most of Luna's mistakes come with…
11 min · 2,468 words
Is 2027 a leap year? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. The answer was No; 80 got it right. See every answer and who dissented.
10 min · 2,222 words
TIS-100 Bench: LLMs Push the Pareto Frontier in Programming Puzzle Games
Wincon gave 16 LLMs the puzzle game TIS-100 and its manual, withheld the community leaderboard, and let them keep optimizing. The models found 133 new Pareto frontier points on 18 of 55 puzzles, including 3 outright cycle records, and players quickly improved on them.
17 min · 3,982 words
Which country has the best universities? what 87 AI models think
We asked 87 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 98% picked United States. See every answer and who dissented.
10 min · 2,335 words
Klang Research sent the same 3,000 sentences through twelve frontier AI models and found every one charges 30% or more extra tokens for Swedish, Norwegian, Danish and Finnish than for English, explaining where the extra tokens come from.
5 min · 1,037 words
The shelf life of LLM findings
Stanford's Owen Queen and James Zou re-ran 50 widely cited findings about language models, from sycophancy and jailbreaks to fabricated citations and long-context limits, across 20 models released July 2024 to September 2026, finding that within 15 months most claims had expired or substantially weakened.
28 min · 6,536 words
Who really runs the group project? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 77% picked The one who does all the work. See every answer and who dissented.
15 min · 3,483 words
Why is Gemini 4 Carbon taking so long?
Kernel traced every step of Gemini 4 Carbon's runs on a private eval to explain why it spent close to an hour on tasks that Opus and Astra finished in minutes, pointing to behavioral issues and mechanical inefficiencies such as fewer tool calls per step and frequent re-reading.
5 min · 1,063 words
Science’s Last Exam: Can AI do Scientific Research on Its Own?
AItonomy introduces Science’s Last Exam (SLE): 113 real research tasks across nine disciplines that test hypothesis discovery, protocol planning, scientific computing, software operation, experiment observation, and end-to-end research—with independent sealed verification.
5 min · 1,080 words
Is it a red flag if they don't have Instagram? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 99% picked No. See every answer, who searched the web, and who dissented.
10 min · 2,294 words
Ryan Madden reports that GPT-5.6 Sol, running in Codex at a low thinking level, completed a peaceful (no biters) Factorio run to rocket launch in 13.5 hours of game time via a programmatic game library, and describes where the model excelled and where late-game scaling broke down.
2 min · 364 words
The one thing every college freshman should know: what 87 AI models think
We asked 87 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 33% picked Office Hours. See every answer and who dissented.
12 min · 2,810 words
Rate the average student's study habits out of 10: what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 45% picked 5. See every answer, who searched the web, and who dissented.
10 min · 2,187 words
MIT or Caltech? what 87 AI models think
We asked 87 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 95% picked MIT. See every answer, who searched the web, and who dissented.
11 min · 2,424 words
The AI cost of JavaScript frameworks
Nolan Lawson has coding agents build the same app five times in each of a dozen JavaScript frameworks, from Next.js and Angular to SvelteKit, Solid, htmx and vanilla Vite, and compares development time, tokens spent, lines of code and Lighthouse scores to argue for evidence-based framework choices.
16 min · 3,730 words
Artificial Analysis v5 and the benchmarker's dilemmaA sort of review of the benchmarking industry and what's missing.
Nathan Lambert reviews Artificial Analysis Intelligence Index v5 and the wider independent-evaluation industry, arguing that every benchmarker must collapse model quality into a single number despite unavoidable aggregation trade-offs, and that judging the benchmarkers is itself more sociological than scientific.
8 min · 1,810 words
Should you drop out to do a startup? what 85 AI models think
We asked 85 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 99% picked No. See every answer, who searched the web, and who dissented.
11 min · 2,509 words
Thomas Neumann argues that Goodhart's law is breaking database benchmarks such as ClickBench, where systems increasingly optimise for the known queries, and proposes keeping benchmark queries secret or generating fresh workloads each round, as SQLStorm does, so results stay meaningful.
3 min · 753 words