Comparing Muon, NorMuon and AdamW for Fine-tuning a Dense Retriever

Qingcheng Zeng gives Muon and NorMuon the same tuning budget as AdamW when fine-tuning a contrastively pretrained dense retriever: lower training loss, no BEIR win. Learning rate and transfer matter more than the optimizer.

Research · Machine Learning · Research · LLMs · Benchmarks

Can a Model Learn New Skills as Add-Ons?

Connito Research trains residual MoE experts with their own routers on a frozen DeepSeek-V2-Lite base, then merges independently trained math, code, medical, law, and finance experts in seconds without retraining—lifting domain benchmarks while leaving the original model untouched.

Research · AI · Machine Learning · Research · LLMs

Mercury 2.5: Intelligence, Performance and Price Analysis

Artificial Analysis profiles Inception's Mercury 2.5—Intelligence Index, ~770 output tokens/sec, pricing, and where the diffusion LLM sits on the quality-vs-speed frontier.

Research · AI · LLMs · Benchmarks · Performance

The plunging price of thought

Epoch AI finds the cost of a given level of AI performance has fallen about 47% per quarter since 2023—roughly 13× per year—faster than DNA sequencing, compute, batteries, or electricity, across math, science, and skill-game benchmarks.

Research · AI · Research · LLMs · Economics

The Function That Beat the Model: What We Measured When We Removed the LLMs

SPERIXLABS replaced a 1B-parameter local model that validated sensitive-data detections with a 40-line Python function, then published the four experiments showing where classical checks beat the LLM on accuracy and latency.

Research · LLMs · Security · Benchmarks · Engineering

modern-fs-benchmark

Bartosz Fenski’s continuous benchmark suite for multi-device CoW filesystems (btrfs, ZFS, bcachefs) measures snapshot aging, compression, rebuild, ENOSPC, and other workloads classic single-disk fio tests miss.

Research · Systems Programming · Benchmarks · Open Source · Infrastructure

Scaling Discovery through Test-Time Communication

Research paper showing that test-time communication among identical agents sharing discoveries can beat independent parallel search on ARC-AGI-3 and transfer to research tasks like polyomino packing and MNIST compression.

Research · AI Agents · Research · Machine Learning · Benchmarks

RTK reports huge token savings, but our cost benchmarks disagree

Quesma ran RTK (Rust Token Killer) against Terminal-Bench 2.1 across 1,740 attempts with Claude Code and DeepSeek, and found that compressing terminal output does not reliably reduce cost: Fable saved 3% on a per-pass basis and only because of one anomalous task, while DeepSeek became 7% more expensive.

Research · AI Coding Agents · LLMs · Benchmarks · Cost Optimization

Why machine learning research agents don't overfit — and what compression has to do with itNew research indicates that AI agents learn compressible models of data, which don't have enough space to enable memorization.

Amazon Science researchers explain why ML research agents fail to overfit benchmarks even after many evaluation rounds, arguing that successful agents learn highly compressible representations that are too compact to store memorised answers — connecting this to Minimum Description Length theory.

Research · Machine Learning · AI Agents · Benchmarks · Research

How well do agents use verification techniques?

Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.

Research · AI Coding Agents · Testing · Formal Methods · Software Quality

GPT-6 Astra on robotic manipulation

Robocurve ran GPT-6 Astra through the same two bimanual robot-arm tasks previously used to benchmark Claude Fable 5 and 5.1. Astra completed the block-into-bowl task in 19 of 20 trials at roughly half the cost per run of Fable 5.1, but matched Fable 5.1's two-out-of-twenty completion rate on the harder puzzle-insertion task.

Research · Robotics · AI · LLMs · Benchmarks

OpenAI's GPT-6 Astra on ARC-AGI-3

The ARC Prize team reports that GPT-6 Astra scored 99.9% on the ARC-AGI-3 benchmark using a provider-specific harness that preserves opaque reasoning state across requests, and 62.7% under a standard provider-neutral harness. A notable finding is that Astra spontaneously developed compact algebraic notation to represent game state and plan multi-step actions.

Research · AI · Benchmarks · AGI · LLMs

Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering

Frontis.AI / Horizon Research open-source OpenMLE (gym, RL, Evo) and Frontis-MA1-35B, lifting MLE-Bench Lite medal average to 71.21% under a single RTX 4090 budget toward executable RSI research.

Research · Machine Learning · Research · AI Agents · Open Source