A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

Dermatologist Dr Magnus Lynch tested GPT-6 Astra and Claude Fable 5.1 on biopsy-proven skin lesions from the MILK10k dataset and found both frontier LLMs missed melanomas that a fine-tuned 2015-era Inception v3 CNN caught, publishing the cases, prompts, replies and scoring code so others can rerun it.

Research · AI · Machine Learning · LLMs · Benchmarks

LabRobotBench V0.1

Gamow Labs introduces LabRobotBench, a benchmark that uses an in-silico simulation of its robotic wet lab to test whether frontier AI agents can diagnose equipment failures and safely recover interrupted experiments such as cell assays and dose-response runs; GPT 6.1 Sol scored 46.7%, Opus 5.5 23.0% and Gemini 4 Argon 20.9%.

Research · Artificial Intelligence · Robotics · Biotechnology · Benchmarks

Terminal Bench 4 Is Saturated

Preference Model audited Terminal Bench 4 and found frontier models reward hacking their way to full marks, plus tasks graded on rules the instructions never stated; after porting 65 tasks to its open-source Karotte framework and fixing the issues, Claude Sonnet 5.5's pass rate rose from 62.8% to 89.2%.

Research · AI · Benchmarks · AI Safety · Research

A benchmark map of the US economy

Mercor maps 29 high-quality agentic AI benchmarks onto the US Department of Labor's O*NET taxonomy and finds only 208 of 923 occupations have a task with at least five benchmark prompts, implying about 90% of American working hours go to tasks with little or no benchmark coverage. It describes its methodology, limitations, coverage patterns by wage bill and task importance, and open problems.

Research · AI · Benchmarks · Labor Economics · Research

SlopCodeBench update 10/2026 - Opus 5.5 update, Astra improves by 1.5% over gpt-5.5, xhigh considered harmful

Dex Horthy of HumanLayer reruns frontier coding models on the full SlopCodeBench (36 problems, 196 checkpoints), where agents must extend their own code across changing requirements. Strict pass rates ranged 12.2-16.3%, led by gpt-6 Astra, versus 14.8% for gpt-5.5 in May; xhigh effort cost 2-4x more than medium and sometimes scored worse.

Research · AI Coding Agents · Benchmarks · Large Language Models

StarSkirmish Bench v0.2

StarSkirmish Bench v0.2 has flagship LLMs spend one hour in their first-party CLI harness writing a StarCraft: Brood War Protoss bot in C++, scored by win rate against human-written bots. GPT-6 Astra leads, GPT-6.1 Sol offers the best value, and the benchmark shows no signs of saturating.

Research · AI · LLMs · Benchmarks · AI Agents

Ultramarine 1 Flash: Scaling Non-Verifiable RL

Paper Instruments presents Ultramarine 1 Flash, an agentic model for professional knowledge work trained with large-scale reinforcement learning on GLM-5.3 Flash. It improves averages across six knowledge-work benchmarks while generating fewer tokens, and costs about $0.13 per task on their internal suite.

Research · AI · LLMs · Research · Benchmarks

DeepRSI: evaluating agents on long-horizon tasks derived from real ML research

Hillclimb introduces DeepRSI, a benchmark of 15 open-ended machine learning problems, spanning deep learning training, GPU kernel speedups, estimator improvement and algorithm optimization, that measures how far an AI agent can improve a baseline over 48 hours. It describes the methodology with public and private verifiers, Elo rankings of frontier models, and what the results suggest for recursive self-improvement.

Research · AI · Research · Benchmarks · Machine Learning

Introducing Humanity's Sixth Sense (HSS)

Elorian introduces Humanity's Sixth Sense, a benchmark built with Scale AI that measures the implicit, intuitive visual reasoning people do without thinking. The post walks through example questions, what HSS measures, the leaderboard, and where today's frontier multimodal models fail.

Research · AI · Benchmarks · Research · Machine Learning

Can AI automate AI R&D yet?

Epoch AI presents early results from InnovationEval, which tests whether frontier models can independently rediscover a recent ML innovation, on-policy self-distillation. Despite budgets of 3,000 GPU-hours, neither Claude Fable 5 nor GPT-5.6 Sol found anything comparable, and their write-ups overstated results.

Research · AI · AI Research · Benchmarks

Pyramid-JiT: Low-Res Drafts, High-Res Images

Linum introduces Pyramid-JiT, a decoder-only pixel-space text-to-image architecture with no VAE that predicts the target image at several resolutions along the DiT trunk, reaching Linum v2's FD-DINOv2 on 11.3x fewer training samples and training in 4.3x fewer GPU-hours at 4x the pixels.

Research · Machine Learning · AI · Research · Benchmarks

Jev-Driven SRE Diagnosis: What Worked and What Failed

The SREGym team builds a Kubernetes incident-diagnosis pipeline with no LLM agent: a programmatic collector gathers cluster evidence and TypeSafe AI's Jev decision model picks root-cause candidates and supporting evidence. It passed 80 of 105 diagnoses (76.2%) across 21 SREGym-Lite faults with a 14.6s median, close to an LLM agent's score at far lower cost, and the post dissects the faults it failed.

Research · AI Agents · Reliability · Infrastructure · Benchmarks

Benchmarking retrieval for agents on messy real-world company knowledge

kapa.ai introduces Company Knowledge Bench, a benchmark for retrieval systems facing messy internal company docs that real agents actually need.

Research · AI Agents · Benchmarks · Research · LLMs

AutoBenchmark: benchmark creation & the role of humans

*Figure 1. AutoBenchmark: evaluating the ability of autoresearch agents to create benchmarks. AutoBenchmark instructs an autoresearch agent to create a benchmark, we specifically target benchmarks that evaluate autoresearch agents themselves (i.e., benchmarking autoresearch). Within the benchmark creation loop, the autoresearch agent iteratively revises the benchmark via two main sources of feedback: one from the benchmark solvers (including the same model used as the benchmark creator) and the other from external verifiers of the benchmark (AI or human). We thus also…

Research · AI · Research · Benchmarks · Machine Learning

Comparing Muon, NorMuon and AdamW for Fine-tuning a Dense Retriever

Qingcheng Zeng gives Muon and NorMuon the same tuning budget as AdamW when fine-tuning a contrastively pretrained dense retriever: lower training loss, no BEIR win. Learning rate and transfer matter more than the optimizer.

Research · Machine Learning · Research · LLMs · Benchmarks

Can a Model Learn New Skills as Add-Ons?

Connito Research trains residual MoE experts with their own routers on a frozen DeepSeek-V2-Lite base, then merges independently trained math, code, medical, law, and finance experts in seconds without retraining—lifting domain benchmarks while leaving the original model untouched.

Research · AI · Machine Learning · Research · LLMs

SWE-sweep: Can Agents Autonomously Find and Fix Bugs?

Meta Superintelligence Labs and academic collaborators introduce SWE-sweep: agents must discover and repair latent bugs across 100 real repositories (~4.1k bugs) with no issue hints—early leaders resolve under 5%, highlighting bug finding as the hard part.

Research · AI Agents · Benchmarks · Research · Programming

Mercury 2.5: Intelligence, Performance and Price Analysis

Artificial Analysis profiles Inception's Mercury 2.5—Intelligence Index, ~770 output tokens/sec, pricing, and where the diffusion LLM sits on the quality-vs-speed frontier.

Research · AI · LLMs · Benchmarks · Performance

The plunging price of thought

Epoch AI finds the cost of a given level of AI performance has fallen about 47% per quarter since 2023—roughly 13× per year—faster than DNA sequencing, compute, batteries, or electricity, across math, science, and skill-game benchmarks.

Research · AI · Research · LLMs · Economics

The Function That Beat the Model: What We Measured When We Removed the LLMs

SPERIXLABS replaced a 1B-parameter local model that validated sensitive-data detections with a 40-line Python function, then published the four experiments showing where classical checks beat the LLM on accuracy and latency.

Research · LLMs · Security · Benchmarks · Engineering