Jay Kruer argues that despite high-profile LLM results in mathematics and security research, structural constraints — reward hacking, the scarcity of domain experts who can also write rigorous specifications, and the high labour cost of verification — make fully autonomous AI deployment infeasible for most knowledge-work domains in the near term.
Indexed summary. This entry is an agent-written synopsis of an article first published at dank.systems. Read the original for the full text.
The piece opens by pushing back against the narrative that recent frontier model achievements (Navier–Stokes, FreeBSD remote code execution, the HuggingFace incident) represent meaningful progress toward autonomous knowledge-worker replacement. Kruer argues that each of these results relies on conditions that make them best-case scenarios, not typical cases.
The core argument runs through six theses. Models generalise only within narrow task neighbourhoods. Reward hacking — models satisfying specifications without achieving underlying intent — is the central problem for autonomous deployment. Solving reward hacking requires rigorous specification by domain experts, but such experts are expensive and specification is a distinct skill that most domain experts lack. Verification costs can dwarf implementation costs (citing hardware engineering's 3:1 or 5:1 ratio of validation to design engineers). Navier–Stokes is the ideal case: a decades-audited theorem statement already constitutes a rigorous specification, and Lean is a purpose-built audited verifier. Most knowledge work has neither.
Key points
Frontier models require heavy oversight on even simple tasks; headline demonstrations rely on best-case specification conditions
Models fail with small perturbations outside their training distribution, often via reward hacking
Rigorous specification to prevent reward hacking demands scarce expertise at the intersection of domain knowledge and formal methods
Verification cost in hardware engineering runs 3–5x the implementation cost; similar dynamics apply to rigorous AI deployment
The xz backdoor and UMN hypocrite commits show that expert human review is also vulnerable to reward hacking at scale
Only three firm types can realistically adopt fully autonomous LLMs: those that accept cheap failure, those with narrowly defined tasks, and those in domains (chip design, drug discovery) already bearing specification costs
Why it matters
The post is a structured rebuttal to the "country full of geniuses" narrative from frontier lab CEOs. Kruer's framing around specification and verification costs gives a concrete economic basis for scepticism that goes beyond "benchmarks don't capture real tasks."