Topic
Everything filed under Benchmarks, newest first.
RSS · JSON · All topics
What is the biggest unsolved problem in science? what 84 AI models think
We put "What is the biggest unsolved problem in science?" to 84 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
19 min · 4,300 words
Automating eval design and hillclimbing with Claude
Lance Martin (claude.dev) explains principles for production-like evals with held-out sets, then shows how the claude-api skill’s build-eval and hillclimb commands automate design and overfitting-aware improvement—including cost and performance case studies.
2 min · 462 words
Can a Model Learn New Skills as Add-Ons?
Connito Research trains residual MoE experts with their own routers on a frozen DeepSeek-V2-Lite base, then merges independently trained math, code, medical, law, and finance experts in seconds without retraining—lifting domain benchmarks while leaving the original model untouched.
4 min · 913 words
Small Decisions: Engineering a Leading Model
AWS engineer Marc Brooker recounts building and training a small leading model hands-on—what worked, how it performed, and what the exercise taught him about modern model-building.
9 min · 2,024 words
What is the date today? what 83 AI models think
We put "What is the date today?" to 83 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
12 min · 2,726 words
Will India win the Asian Games cricket gold? 80 AI models predict the result
We put "Will India win the Asian Games cricket gold?" to 80 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
15 min · 3,517 words
Ravens vs Cowboys in Rio de Janeiro: 85 AI models predict the result
We put "Ravens vs Cowboys in Rio de Janeiro" to 85 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
16 min · 3,627 words
Harvard or Stanford? what 84 AI models think
We put "Harvard or Stanford?" to 84 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
16 min · 3,568 words
India vs West Indies, 1st ODI: 81 AI models predict the result
We put "India vs West Indies, 1st ODI" to 81 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
13 min · 3,103 words
Analyzing Frontier Model Progress with My Favourite Game: Prince of Persia (Apple II, 1989)
Priyan uses Jordan Mechner's Prince of Persia as a living benchmark: asking frontier models to port and reason about the classic Apple II game, and what those runs reveal about coding-agent progress.
8 min · 1,825 words
Who wins the Azerbaijan Grand Prix? 82 AI models predict the result
We put "Who wins the Azerbaijan Grand Prix?" to 82 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
13 min · 2,991 words
Jordan or LeBron? what 85 AI models think
We put "Jordan or LeBron?" to 85 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
16 min · 3,606 words
Who wins the India v West Indies ODI series? 82 AI models predict the result
We put "Who wins the India v West Indies ODI series?" to 82 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
14 min · 3,178 words
“When a measure becomes a target, it ceases to be a good measure” – Goodhart’s law Current AI research, especially the frontier LLM research, is dominated by benchmarks. It is the first thing we look at when a new model comes out, it is the headline of each release, and they dominate the discourse when […]
13 min · 2,940 words
Which AI is the most annoying? what 84 AI models think
We asked 84 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 12% picked Grok Edginess. See every answer and who dissented.
11 min · 2,455 words
SWE-sweep: Can Agents Autonomously Find and Fix Bugs?
Meta Superintelligence Labs and academic collaborators introduce SWE-sweep: agents must discover and repair latent bugs across 100 real repositories (~4.1k bugs) with no issue hints—early leaders resolve under 5%, highlighting bug finding as the hard part.
6 min · 1,411 words
What is the most useless college major? what 84 AI models think
We asked 84 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 17% picked General Studies. See every answer and who dissented.
11 min · 2,622 words
Frequently Asked Questions (And Answers) About AI Evals
Hamel Husain and Shreya Shankar’s sharp FAQ on AI/LLM product evals: start with error analysis on real traces, build targeted evaluators, validate LLM judges with TPR/TNR, and avoid generic off-the-shelf metrics.
74 min · 17,016 words
How we evaluate AI assistants at Studio Jadu
Studio Jadu’s Miquel Farré explains how the animation studio builds, evaluates, and keeps control of AI assistants as prompts, models, tools, and conversations change—beyond shipping a first demo.
9 min · 2,133 words
Mercury 2.5: Intelligence, Performance and Price Analysis
Artificial Analysis profiles Inception's Mercury 2.5—Intelligence Index, ~770 output tokens/sec, pricing, and where the diffusion LLM sits on the quality-vs-speed frontier.
12 min · 2,677 words