How well do agents use test/verification techniques?
TL;DR Highlight
Naming test techniques to coding agents didn't beat default; they apply them superficially
Who Should Read
Backend developer who tells coding agents to use TDD or fuzzing and wants to know whether that actually improves code correctness
Core Mechanics
Community Reactions
- Reproducibility concerns
- One commenter said the experiment is nearly worthless without the exact prompts and test setup.
- Another asked whether a repo exists to reproduce it.
- Audit and cross-model review worked
- Several commenters reported good results auditing code with LLMs from different families.
- One used adversarial review in a fresh context with good results.
- TDD and Hypothesis stay shallow
- One commenter found the Superpowers skill's TDD produced low-value tests like checking a button exists.
- Another found agents struggled to connect Hypothesis tests to business rules and obsessed over floating-point edge cases.
- Verus worked once specs were solid
- One commenter said agents left alone write specs that just restate the implementation.
- After careful spec work, Claude proved the code and found a few bugs.
- Automate mutation testing instead
How to Apply
- Add mutation checks to agent-written tests
- Run cargo-mutants or your language's mutation tool on one module today.
- Count surviving mutants to see whether the tests catch code changes.
- Tell agents what to stop doing, not a technique name
- Replace "use fuzzing" with rules like no repeated identical inputs and no invalid-only inputs.
- This mirrors the author's skill that nudged away from default behavior and did OK.
- Audit with a separate context
- After implementation, have a different model in a fresh context audit the code and list bugs.
- Feed that list into a fix pass.
- Skip long tutorial-style skills
- Shorten any skill over 20k tokens and compare against Default.
- Remove it if correctness doesn't improve.
Try It
# Paste into Claude Code or Codex (simplified test-audit prompt based on the article's findings)
Audit the tests of {{module}}.
- Find tests that feed the same value to multiple inputs (e.g. all streams identical).
- Find tests with symmetric/palindromic inputs that can't catch ordering or swap bugs.
- Find always-true assertions (A => A) or property tests that only hit invalid inputs.
Rewrite each bad test, then inject a deliberate bug to confirm the test fails, and revert it.
Next: run this in a fresh session separate from the one that wrote the code and compare results.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.