RTK reports huge token savings, but our cost benchmarks disagree

Quesma ran RTK (Rust Token Killer) against Terminal-Bench 2.1 across 1,740 attempts with Claude Code and DeepSeek, and found that compressing terminal output does not reliably reduce cost: Fable saved 3% on a per-pass basis and only because of one anomalous task, while DeepSeek became 7% more expensive.

Research · AI Coding Agents · LLMs · Benchmarks · Cost Optimization

How well do agents use verification techniques?

Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.

Research · AI Coding Agents · Testing · Formal Methods · Software Quality