Topic
Everything filed under Testing, newest first.
RSS · JSON · All topics
Kyle Kingsbury announces hegel-clj, a Clojure interface to the Hegel generative testing library, and explains the frustrations with test.check that motivated it and how Hegel's design improves generative testing.
9 min · 2,116 words
Designing a Multi-Agent Architecture Guided by Advanced Type Systems
An adversarial multi-agent design that splits development across Specifier, Implementer, Falsifier and Verifier agents, using advanced type systems in Python and TypeScript as proof engines to curb silent LLM hallucinations.
11 min · 2,578 words
Kyle Kingsbury's Jepsen analysis of the Orkes Conductor workflow orchestrator (versions 5.3.2 to 5.5.0-rc2) found frequent concurrent execution of individual tasks, a delayed-execution issue, four minor startWorkflow errors, and three serious bugs that abandoned, forgot or falsely completed workflow executions. Orkes 5.5.6 patches all serious safety issues. Funded by Orkes.
22 min · 5,043 words
AI Can Write the Code. Who Verifies It?
Mihai Avram argues that AI coding agents have outpaced human code review, citing Faros data on rising bugs, review time and incidents, and lays out layered automated verification: AI code reviewers (Claude, Codex, Copilot, Bugbot), CI/CD checks from lint, tests and SAST to architecture review, and when to run them from pre-commit to production.
16 min · 3,785 words
Introducing the Accessible Components Library
TetraLogical's Craig Abbott introduces its open-sourced Accessible Components Library 1.0.0: eight components (accordion, breadcrumbs, dialog, disclosure, pagination, tabs, text input, textarea) with HTML, CSS and JavaScript, plus documentation of accessibility requirements, test scenarios, assistive technology compatibility and working automated tests, and a discussion of what automated tests can and cannot check.
8 min · 1,795 words
Some quick thoughts on Unit Testing ActivityPub
As part of his NLnet-funded work on the ActivityBot framework, Terence Eden describes the problems he has hit writing unit tests for ActivityPub: validating usernames and text, validating messages, verifying HTTP signatures, and testing communication with other servers. He is publishing his tests and a collection of sample messages and signatures for other projects.
2 min · 524 words
The tests your agent writes for itself
Mashhood Rastgar responds to Kun Chen's DeepSWE v1.1 experiment, where banning Claude Sonnet 5.5 from writing its own tests left success flat while cutting time and cost, arguing the real lesson is about unrequested agent-written tests versus human-specified tests that encode intent.
3 min · 768 words
When random is not actually random enough
Austin Seipp of East River Source Control shows why picking an item with random_u64() modulo n skews the distribution, and argues for APIs that express the desired distribution directly, such as weighted random_choice, which his team now uses everywhere in its Antithesis-related test code.
8 min · 1,816 words
Property Testing with Agent Swarms
Inanna Malick shows how to have coding agents build property-testing machinery (reference models, generators and assertions) for a repo, then turn the failures into small, reviewable fixes. Pointed at OpenAI's Codex, the public playbook found thirteen distinct correctness bugs, and runs on jj, uv, Prometheus and Babel produced merged upstream fixes.
6 min · 1,325 words
Stefano Saitta pushes back on the claim that coding agents kill unit tests, arguing for a pragmatic mix of E2E, integration, and still-useful unit tests in an AI-assisted workflow.
4 min · 839 words
Deterministic simulation testing in celld: exploring execution orders and reproducing bugs
Yusuke Tanaka explains how the Deno team uses deterministic simulation testing for celld, a self-hostable runtime for Cloudflare Workers and Durable Objects apps, to explore message orderings, storage failures and node restarts, replay failing runs exactly, and turn discovered bugs into regression tests.
6 min · 1,426 words
How well do agents use verification techniques?
Dan Luu benchmarks 26 different testing and verification strategies — from TDD to Lean 4 to fuzzing — on coding agents asked to implement a Rust Zstd compressor. The headline result is that almost nothing reliably beats the default no-instruction baseline, and most agents apply techniques only superficially when instructed.
1 min · 287 wordsagent-written