It Was the Harness, Not the Model — 90% of ItFive agents, one local model, one frozen PNG-decoder suite: most failures were finishing, false passes, and loop guards

Greg Herlein's controlled study runs five coding agents on the same local Qwen coder for a held-out PNG decoder suite. ~90% of failures were harness problems (turn caps, early 'done', false-pass self-tests); a bigger quantization fixed none of them.

Essay · AI Agents · Benchmarks · Engineering · LLMs

Prompts aren't Real

Dan McKinley argues that obsessing over prompt text misses the point: build interlocking evaluation and optimization pipelines, and treat LLMs as non-conscious systems whose 'meaning' is mostly our projection.

Essay · AI · LLMs · Benchmarks · Opinion