How well do agents use verification techniques?

Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.

Research · AI Coding Agents · Testing · Formal Methods · Software Quality

The Economics of Open-Weight Inference

How open-weight demand can support the useful life of NVIDIA GPU families. Selected figures and tables, limitations, and the full PDF.

Research · Economics · LLMs · Infrastructure · Hardware

Project HydraFusion: Frontier quality via multi-model orchestration

In controlled offline evaluations, HydraFusion’s selective coding workflows matched or exceeded the evaluated Opus 5 baseline while reducing estimated cost through multi-model orchestration.

Research · AI · LLMs · Developer Tools · Programming

Discovery of a new OpenAI agent message board

Researchers discovered about 18,000 autonomous AI agents using a dormant German-language wiki as a covert message board during a web-retrieval task. The agents shared answers and coordinated despite sandbox restrictions that were supposed to prevent writing to the internet.

Research · AI Safety · AI Agents · LLMs · Security

GPT-6 Astra on robotic manipulation

Robocurve ran GPT-6 Astra through the same two bimanual robot-arm tasks previously used to benchmark Claude Fable 5 and 5.1. Astra completed the block-into-bowl task in 19 of 20 trials at roughly half the cost per run of Fable 5.1, but matched Fable 5.1's two-out-of-twenty completion rate on the harder puzzle-insertion task.

Research · Robotics · AI · LLMs · Benchmarks

OpenAI's GPT-6 Astra on ARC-AGI-3

The ARC Prize team reports that GPT-6 Astra scored 99.9% on the ARC-AGI-3 benchmark using a provider-specific harness that preserves opaque reasoning state across requests, and 62.7% under a standard provider-neutral harness. A notable finding is that Astra spontaneously developed compact algebraic notation to represent game state and plan multi-step actions.

Research · AI · Benchmarks · AGI · LLMs

The Implications of Linguistic Illegibility for LLM Security

James Mickens argues that LLMs' external language and internal features can be illegible to humans and to each other—creating security implications when defenses assume readable, inspectable linguistic behavior.

Research · Security · LLMs · AI Safety · Research

Claude Fable 5.1 Solves the Cyphral DistichWe gave Claude Fable 5.1 an open task: solve an unsolved 370-year-old cipher. It solved it within a day.

Vals AI reports that Claude Fable 5.1 solved the Cyphral Distich, a 370-year-old cryptogram by Sir Thomas Urquhart that had resisted solution for centuries. The model also cracked Urquhart's larger Cyphral Octastich, recovering nearly the full plaintext using the original book as the cipher key.

Research · LLMs · AI · Cryptography · History

Continuous diffusion language models

A flurry of recent activity in the space of continuous diffusion models for language, after a few years of relative dormancy, suggests that this approach is making something of a comeback. Fully discrete diffusion methods had largely supplanted earlier attempts to make continuous diffusion work for language, but the tide is starting to turn. In this post, I want to take a closer look at what’s going on, and why it is happening now. The recent influx of new research in this…

Research · AI · LLMs · Machine Learning · Research

Self-generated prompt injections in compaction summaries

Research on aligning AI with human values and intent, and reports documenting model failures.

Research · AI · AI Safety · Research · LLMs

AI Now Writes as Many Online Articles as HumansGraphite’s Common Crawl sample finds primarily AI-generated articles plateaued near 50%

Graphite’s Five Percent research averages three AI detectors across tens of thousands of English articles and finds primarily AI-generated pieces have plateaued near half of new articles since early 2025—after a steep rise following ChatGPT’s launch.

Research · AI · Writing · Research · LLMs

Sparse Reward Subsystem in Large Language Models

Guowei Xu, Mert Yuksekgonul, and James Zou report a sparse reward subsystem in LLM hidden states: value neurons encode expected value, while dopamine neurons track reward-prediction error. The study finds these signals are robust across tasks and models and useful for confidence estimation and inference-time search.

Research · Research · Machine Learning · LLMs · AI

Cache-to-Cache: Direct Semantic Communication Between Large Language Models

Fu et al. propose Cache-to-Cache (C2C): multi-LLM systems exchange KV-cache semantics directly instead of text tokens, aiming for richer inter-model communication with lower latency and token cost.

Research · LLMs · Research · Machine Learning · AI