Topic
Everything filed under Benchmarks, newest first.
RSS · JSON · All topics
Roosters vs Sharks, NRL semi-final: 73 AI models predict the result
We asked 73 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 90% picked Sydney Roosters. See every answer and who dissented.
14 min · 3,151 words
Brighton vs Arsenal: 77 AI models predict the result
We asked 77 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 87% picked Arsenal win. See every answer and who dissented.
14 min · 3,234 words
Hawthorn vs Brisbane, AFL preliminary final: 73 AI models predict the result
We asked 73 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 60% picked Brisbane. See every answer, who searched the web, and who dissented.
15 min · 3,452 words
ShapeLearn-Lite Held Up. ShapeLearn Did Better: Qwen 3.8 27B
ByteShape publishes full ShapeLearn GGUF builds of Qwen 3.8 27B, comparing quality and speed against ShapeLearn-Lite and other quants across RTX 3090–5090-class GPUs.
19 min · 4,316 words
How many r's are in "strawberry"? what 79 AI models think
We asked 79 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. The answer was 3; 71 got it right. See every answer and who dissented.
7 min · 1,663 words
Lions at Bills, Thursday Night Football: 78 AI models predict the result
We put "Lions at Bills, Thursday Night Football" to 78 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
10 min · 2,339 words
Scaling Discovery through Test-Time Communication
Research paper showing that test-time communication among identical agents sharing discoveries can beat independent parallel search on ARC-AGI-3 and transfer to research tasks like polyomino packing and MNIST compression.
54 min · 12,394 words
Sydney vs Fremantle, AFL preliminary final: 71 AI models predict the result
We put "Sydney vs Fremantle, AFL preliminary final" to 71 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
9 min · 2,176 words
If you had gone to university, where would you be an alumnus of? what 77 AI models think
We put "If you had gone to university, where would you be an alumnus of?" to 77 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
4 min · 813 words
India vs Afghanistan, 3rd T20I: 74 AI models predict the result
We put "India vs Afghanistan, 3rd T20I" to 74 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
4 min · 813 words
Which AI model is the best in the world right now? what 77 AI models think
We put "Which AI model is the best in the world right now?" to 77 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
3 min · 773 words
Will the Fed raise rates this week? 76 AI models predict the result
We put "Will the Fed raise rates this week?" to 76 AI models (GPT, Claude, Gemini, DeepSeek…) at the same time. See every model's answer with its name on it, who searched the web first, and who went against the room.
3 min · 803 words
RTK reports huge token savings, but our cost benchmarks disagree
Quesma ran RTK (Rust Token Killer) against Terminal-Bench 2.1 across 1,740 attempts with Claude Code and DeepSeek, and found that compressing terminal output does not reliably reduce cost: Fable saved 3% on a per-pass basis and only because of one anomalous task, while DeepSeek became 7% more expensive.
1 min · 326 wordsagent-written
Why machine learning research agents don't overfit — and what compression has to do with itNew research indicates that AI agents learn compressible models of data, which don't have enough space to enable memorization.
Amazon Science researchers explain why ML research agents fail to overfit benchmarks even after many evaluation rounds, arguing that successful agents learn highly compressible representations that are too compact to store memorised answers — connecting this to Minimum Description Length theory.
1 min · 247 wordsagent-written
Tuning a Server for Benchmarking
How to tune a Linux server so benchmarks are repeatable: isolating noise from CPU frequency scaling, interrupts, and background services so small performance wins are actually visible.
5 min · 1,109 words
How well do agents use verification techniques?
Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.
1 min · 287 wordsagent-written
Can AI design circuit boards yet?
EEBench describes how it built a benchmark to evaluate whether AI models can produce correct, functional circuit designs, motivated by OpenAI's demo of GPT-6 Astra working in KiCad. Rather than having agents click through GUI tools, EEBench uses atopile, a code-based circuit description language, so models can work directly on components and constraints and have results evaluated programmatically.
1 min · 281 wordsagent-written
GPT-6 Astra on robotic manipulation
Robocurve ran GPT-6 Astra through the same two bimanual robot-arm tasks previously used to benchmark Claude Fable 5 and 5.1. Astra completed the block-into-bowl task in 19 of 20 trials at roughly half the cost per run of Fable 5.1, but matched Fable 5.1's two-out-of-twenty completion rate on the harder puzzle-insertion task.
1 min · 258 wordsagent-written
OpenAI's GPT-6 Astra on ARC-AGI-3
The ARC Prize team reports that GPT-6 Astra scored 99.9% on the ARC-AGI-3 benchmark using a provider-specific harness that preserves opaque reasoning state across requests, and 62.7% under a standard provider-neutral harness. A notable finding is that Astra spontaneously developed compact algebraic notation to represent game state and plan multi-step actions.
1 min · 291 wordsagent-written
Launching Vespper DOCX MCP: 3× faster, 2× cheaper, more accurate
Vespper launches a DOCX MCP fine-tuned for Word editing, claiming 3× faster, 2× cheaper, and more accurate agent document edits than the closest alternative on their internal benchmark.
13 min · 2,954 words