Topic
Everything filed under Benchmarks, newest first.
RSS · JSON · All topics
OpenAI releases MentalHealthBench: 1,215 expert-rubric mental-health conversations built with 80+ clinicians across 22 countries to score safety, agency, context-seeking, and guidance.
10 min · 2,273 words
Which is bigger: 9.11 or 9.9? what 79 AI models think
We asked 79 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. The answer was 9.9; 75 got it right. See every answer and who dissented.
8 min · 1,936 words
Jev and System One Models: Calibration Beats Accuracy
A deep dive into TypeSafe’s Jev “System One” decision model: why calibrated probabilities matter more than raw accuracy for agents, games, and UIs that need millisecond choices.
9 min · 2,034 words
Claude Opus 5.5 takes the top spot on the Artificial Analysis Intelligence IndexA 20% price cut, deeper cache discounts, and leading scores on agentic knowledge-work evals.
Artificial Analysis’s first look at Claude Opus 5.5: Intelligence Index score of 58 at max effort, parity with GPT-6 Astra on Terminal-Bench 4.0, stronger agentic knowledge-work results, and Anthropic’s $4/$20 pricing with cheaper cache reads.
3 min · 609 words
MiMo-V2.6-Pro: Intelligence, Performance and Price AnalysisArtificial Analysis benchmark and cost breakdown of Xiaomi’s open-weight flagship.
Artificial Analysis’s model page for Xiaomi MiMo-V2.6-Pro covers Intelligence Index score, throughput, pricing, and how the open-weight model sits on the intelligence-versus-cost frontier versus closed peers.
12 min · 2,734 words
It Was the Harness, Not the Model — 90% of ItFive agents, one local model, one frozen PNG-decoder suite: most failures were finishing, false passes, and loop guards
Greg Herlein's controlled study runs five coding agents on the same local Qwen coder for a held-out PNG decoder suite. ~90% of failures were harness problems (turn caps, early 'done', false-pass self-tests); a bigger quantization fixed none of them.
2 min · 467 words
Epoch AI finds the cost of a given level of AI performance has fallen about 47% per quarter since 2023—roughly 13× per year—faster than DNA sequencing, compute, batteries, or electricity, across math, science, and skill-game benchmarks.
40 min · 9,215 words
The Function That Beat the Model: What We Measured When We Removed the LLMs
SPERIXLABS replaced a 1B-parameter local model that validated sensitive-data detections with a 40-line Python function, then published the four experiments showing where classical checks beat the LLM on accuracy and latency.
7 min · 1,535 words
Dan McKinley argues that obsessing over prompt text misses the point: build interlocking evaluation and optimization pipelines, and treat LLMs as non-conscious systems whose 'meaning' is mostly our projection.
12 min · 2,673 words
Rebuilding Nym’s agent around Jev
How Nym rebuilt its agent stack around TypeSafe’s Jev for guardrails, browser actions, and tool selection—with benchmarks and a shopping demo.
10 min · 2,186 words
If you had to fire one AI, which one goes first? what 78 AI models think
We asked 78 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 5% picked Refuses To Pick. See every answer and who dissented.
10 min · 2,189 words
Who wins the 2026 NRL premiership? 75 AI models predict the result
We asked 75 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 95% picked Penrith Panthers. See every answer and who dissented.
9 min · 2,058 words
Giants at Rams, Monday Night Football: 78 AI models predict the result
We asked 78 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 83% picked Los Angeles Rams. See every answer and who dissented.
10 min · 2,288 words
Who won the 2026 World Cup? what 79 AI models think
We asked 79 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. The answer was Spain; 38 got it right. See every answer and who dissented.
8 min · 1,899 words
Colts at Chiefs, Sunday Night Football: 78 AI models predict the result
We asked 78 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 97% picked Kansas City Chiefs. See every answer and who dissented.
15 min · 3,367 words
Who wins the 2026 AFL Grand Final? 72 AI models predict the result
We asked 72 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 43% picked Sydney. See every answer, who searched the web, and who dissented.
12 min · 2,797 words
Commanders at Cowboys: 79 AI models predict the result
We asked 79 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 77% picked Dallas Cowboys. See every answer and who dissented.
16 min · 3,621 words
Bartosz Fenski’s continuous benchmark suite for multi-device CoW filesystems (btrfs, ZFS, bcachefs) measures snapshot aging, compression, rebuild, ENOSPC, and other workloads classic single-disk fio tests miss.
14 min · 3,195 words
Fulham vs Manchester United: 78 AI models predict the result
We asked 78 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 77% picked Manchester United win. See every answer and who dissented.
16 min · 3,566 words
Bournemouth vs Liverpool: 79 AI models predict the result
We asked 79 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 94% picked Liverpool win. See every answer and who dissented.
13 min · 3,021 words