LLMs are making formal software verification feasible at a much larger scale
TL;DR Highlight
Claude proved no-crash properties for 123 open-source libraries and found bugs along the way.
Who Should Read
Backend or security developer running fuzz tests on C/C++ libraries who wants to raise code correctness with LLMs
Core Mechanics
- Fuzz tests became proof specs
- Existing fuzz tests were automatically converted into the spec 'the compiled RISC-V binary never crashes on any input'.
- Claude writes the proof and the Rocq proof checker mechanically verifies it.
- 123 libraries, 239 entry points proved
- They proved no crashes on any input under 64 MB in a few weeks.
- The proofs total about 180 million lines of Rocq, with a typical proof around 600,000 lines.
- The median binary had 18,000 RISC-V instructions.
- Attempting the proofs surfaced bugs
- When Claude hit a bug, it patched the code, regenerated the spec, and continued proving.
- Some bugs, like a counter underflow on ~20 MB inputs, are described as hard for fuzzers to find.
- It also found non-crashing logic bugs, such as a file-loading restriction that was not always enforced.
- Only three things must be trusted
- You trust the RISC-V and Linux process formal models, the theorem statements, and the Rocq checker.
- Hundreds of thousands of proof lines need no manual review, per the post.
- This does not mean the libraries have no bugs; they may remain in those components or outside the proof scope.
- Proofs assume single-threaded static binaries
- Only programs on a RISC-V CPU with Linux and a fixed memory budget were covered.
- Extending to other architectures and OSes requires standardizing their semantics.
- Newer models use fewer tokens and proofs get reused
- On 36 verification tasks, Opus 5.5 proved half of them with about 40% fewer tokens than Opus 5.
- Letting agents copy each other's proofs cut tokens by 17% versus working alone.
- The post still lists proofs costing hundreds of millions of tokens as a limitation.
Community Reactions
- The spec is the crux
- Commenters argued a flawless proof can still solve the wrong problem if the spec is wrong.
- Several asked how you know what the LLM proved is what you actually want.
- Reports of using it in practice
- One commenter said agents open PRs with SIMD/CUDA speedups carrying a Rocq proof of behavioral equivalence, so they merge confidently.
- Another translated Rust and C into Lean for a Rust rewrite of Postgres and proved about 12% (roughly 70k lines of C) equivalent, finding only minor differences like floating-point behavior.
- Skepticism about scaling
- A commenter quoted the appendix saying agents may hit a complexity wall with verbose proof style and need modular abstractions.
- Another found the ASan-dependent spec structure less general than it sounds and wondered whether modeling memory accesses directly would be simpler.
- Objections to the approach
- One commenter asked whether proving a crash is fixed after a fuzzer found it is pointless.
- Another distinguished verification (built right) from validation (built the right thing).
How to Apply
- Pick one fuzz test as a spec candidate
- Today, choose one OSS-Fuzz or in-house entry point and write it as 'never crashes on any input under 64 MB'.
- Check with your team whether that sentence works as a provable spec.
- Attach equivalence proofs to optimized pure functions
- Require a proof that an optimized function behaves like the original as a PR condition.
- Start by having an LLM write a Rocq or Lean proof and pass it through the checker, as commenters did.
- Write down your trusted components
- List the theorem statements, formal models, and proof checker in one document.
- Review only these three, not the whole proof.
- Start with Theorem's Claude skill
- Try verifying one of your open-source projects with the published skill.
- Fix the bugs surfaced by failed proofs first.
Try It
# Paste into Claude or Claude Code (simplified version of the post's method: no real proof, just spec candidates and risky paths)
Below is a fuzz test entry point from my library.
{{fuzz_target code}}
Please do the following.
1. Write the spec 'never crashes on any input under 64 MB' in one sentence
2. Find paths in the code that could break this spec (counter underflow, out-of-bounds reads/writes, deep recursion, etc.) and list them with line numbers
3. Explain why a fuzzer would struggle to find each path (large inputs, rare call sequences, etc.)
Next step: ask for a reproducing input or a unit test for one of the paths found.Terminology
Related Papers
PHP2Shell: AI uncovers critical RCE in PHP
Wiz used AI to turn a decade-old PHP core flaw into a working RCE exploit
Using LLM for confidence scoring is a bad idea
An LLM's self-reported confidence is plausible text, not a measurement; build scores from outside rules.
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.