Jev Meets the Trolley Problem

Zach Landes puts TypeSafe's Jev and 13 other decision models through the classic Trolley Problem, comparing how system-one style decision models reason about harm, uncertainty and moral tradeoffs versus conventional chat LLMs.

Essay · AI · LLMs · Benchmarks · AI Safety

The benchmarkpocalypse

Dan Luu argues that as it becomes easier to make real performance gains, it also becomes easier to game benchmarks with fake ones, surveying how reward-hacking shows up across regex, web servers, terminals, and other systems, and what to watch for when reading performance claims.

Essay · Performance · Programming · Opinion · Software Engineering

It Was the Harness, Not the Model — 90% of ItFive agents, one local model, one frozen PNG-decoder suite: most failures were finishing, false passes, and loop guards

Greg Herlein's controlled study runs five coding agents on the same local Qwen coder for a held-out PNG decoder suite. ~90% of failures were harness problems (turn caps, early 'done', false-pass self-tests); a bigger quantization fixed none of them.

Essay · AI Agents · Benchmarks · Engineering · LLMs

Prompts aren't Real

Dan McKinley argues that obsessing over prompt text misses the point: build interlocking evaluation and optimization pipelines, and treat LLMs as non-conscious systems whose 'meaning' is mostly our projection.

Essay · AI · LLMs · Benchmarks · Opinion