Software Quality

How well do agents use verification techniques?

Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.

Research · AI Coding Agents · Testing · Formal Methods · Software Quality