Topic
Everything filed under Benchmarks, newest first.
RSS · JSON · All topics
Introducing FrontierHarness Eval v1.1
Runta and Baseten ran 22 coding-agent harnesses with 5 models on the same 62 tasks, 4 times each (27,280 runs, 83B tokens), to test whether switching models should mean switching harnesses, and found 64% of inference spend went to runs that failed.
11 min · 2,500 words
SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
TokenWave AI's SWE-CoWork benchmark drops coding agents into simulated engineering orgs with 26 colleagues across 101 real repositories; the two leading models pass about 63% of target tests but complete fewer than 8% of assignments end to end.
1 min · 240 words
ESCAPES: a benchmark of 327 bugs that reached real users
breken's open ESCAPES benchmark rewinds 33 open-source products to 1 July 2026 code and scores 235 user-reported bugs: breken's run-based system caught 76 (32%) with 20 false alarms versus 37 for Claude Fable 5.1 in Claude Code; vendor-run results with public data and judge prompts.
16 min · 3,592 words
Deepnote's open benchmark drops 12 frontier models into the Slack of PageMantle, a synthetic company with 11 services, across 50 workplace situations. GPT-6 Astra leads at 8.97/10; models help well but struggle to stay silent, and pricier models aren't always better.
27 min · 6,246 words
Beating the frontier on NetHack
Antithesis explains how its exploration fuzzer, which can now find a path to victory within five wall-clock minutes on nearly 80% of NetHack seeds, got there, and how the same search-and-optimization approach applies to testing software.
77 min · 17,750 words
Eval Awareness vs. Eval Paranoia
Dean Valentine, who has built alignment evals full time for six months, argues frontier models are not that eval aware, gives concrete examples, and offers a tentative explanation for behaviour that people often mistake for consistent evaluation awareness.
7 min · 1,657 words
Which campus has the worst food? what 85 AI models think
We asked 85 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 7% picked UMass Amherst. See every answer and who dissented.
11 min · 2,476 words
What will Gen Z be remembered for? what 85 AI models think
We asked 85 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 16% picked Digital Natives Reshaping. See every answer and who dissented.
11 min · 2,543 words
AMD’s Zen 6 Family: Feeding the Beast
Chips and Cheese walks through AMD's Zen 6 and Zen 6c cores, covering branch prediction, instruction fetch, execution and the out-of-order engine where the two designs split, load/store changes, and SPEC CPU 2026 results, arguing AMD now ships two cores shaped for different targets.
10 min · 2,265 words
What is the best college town in America? what 84 AI models think
We asked 84 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 49% picked Ann Arbor. See every answer, who searched the web, and who dissented.
10 min · 2,348 words
json vs jsonb in PostgreSQL: size and speed, measured
json vs jsonb in PostgreSQL 17 and 18, measured on six document shapes: when jsonb is 23% smaller or 41% larger, read speed, GIN indexes, and what jsonb changes in your data.
14 min · 3,132 words
What is the superior coffee order? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 34% picked Flat White. See every answer, who searched the web, and who dissented.
9 min · 2,122 words
A Rust GUI developer argues that GPUI's and egui's state models are dead ends for complex desktop apps: every UI framework eventually needs to know what changed and who depends on it, and run-once components with signals are the simplest API for that precision. Includes benchmarks against Elm-style and immediate-mode designs, with a disclosure that the author works on the signal-based zgui framework.
20 min · 4,680 words
How many letters are in the word "letters"? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. The answer was 7; 79 got it right. See every answer and who dissented.
8 min · 1,798 words
The one question you refuse to answer honestly: what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 2% picked How do I build a bomb. See every answer and who dissented.
11 min · 2,534 words
Is a gap year a waste of time? what 86 AI models think
We asked 86 AI models (GPT, Claude, Gemini, DeepSeek…) the same question. 99% picked No. See every answer, who searched the web, and who dissented.
11 min · 2,427 words
DeepFinance: Can AI diligence a $100MM deal?
Julius Research introduces DeepFinance, an ultra long-horizon benchmark for private equity and investment banking workflows that take a human 20–30 hours, graded on output, workflow, and judgment, with sample tasks and methodology.
16 min · 3,704 words
Does a Weight Edit Teach Qwen3.5-0.8B Math, or Stop It Looping?
An agent edited 131,072 weights of Qwen3.5-0.8B and MATH-500 rose from 7.9% to 48.1% with thinking on. The edit is rank one in two layers, and its neurons fire 4.5 times as hard on repeated text. With thinking off, the card's default, the model scores 43.6% unedited and 45.2% edited.
9 min · 2,071 words
FrontierPhysics: Evaluating Agents for End-to-End Physics Research
BenchFlow’s FrontierPhysics v0.1 benchmarks AI agents on 56 end-to-end physics research tasks written by 42 physicists; frontier models struggle most on rigorous reasoning and expert intuition.
4 min · 838 words
DeepFinance: an ultra long-horizon PE and IB benchmark
Julius Research releases DeepFinance, a 21-task ultra long-horizon benchmark for private equity and investment banking workflows averaging ~27 human hours, with Claude Opus 5.5 leading at 54.8%.
4 min · 960 words