Toward automated verification of unreviewed AI-generated code

Mar 16, 2026•peterlavigne•View Original

TL;DR Highlight

An experiment in trusting AI-generated code without reading a single line — combining property-based testing and mutation testing to verify correctness automatically. An interesting attempt to shift code review from 'reading' to 'verifying,' though it only works for simple FizzBuzz-level problems.

Who Should Read

Developers who want to adopt AI coding agents in real work but worry about code quality and safety — especially teams designing workflows for deploying AI-generated code to production, with interest in test automation.

Core Mechanics

Property-based testing (generating many random inputs and checking invariants hold) is more effective at catching edge cases than hand-written unit tests for AI-generated code.
Mutation testing (deliberately introducing bugs into code and verifying the test suite catches them) is useful for measuring how thoroughly tests cover the generated code.
The combination of the two dramatically reduces the need to manually read generated code — the author claims you can trust correctness purely through automated verification for simple algorithmic problems.
The critical limitation: this approach only works for problems with clear, mathematically definable invariants (like FizzBuzz). Real-world business logic with complex state and side effects is much harder to cover with properties.
The author acknowledges this is more of a proof-of-concept than a production-ready workflow — it shows the direction but requires significant additional engineering for practical use.

Evidence

Commenters broadly agreed with the direction but pointed out the 'hard part': defining good properties is itself a skill that requires understanding the problem domain, which means you can't fully escape the need to understand the code.
Several noted that property-based testing is underutilized in general and this is a good reminder of its value — regardless of whether you use it with AI-generated code.
The mutation testing part drew skepticism: running mutation tests on non-trivial code is very slow, making it impractical for rapid iteration cycles.
A comment argued this is essentially the same challenge as formal verification — useful in theory but expensive to apply broadly. The value-to-cost ratio needs to improve before it sees wide adoption.

How to Apply

For pure algorithmic functions (sorting, parsing, calculations), try property-based testing libraries (Hypothesis for Python, fast-check for JS) to validate AI-generated implementations.
Use mutation testing tools (mutmut, Stryker) periodically — not on every commit — to audit test suite quality for critical paths.
When using AI to generate code, have it also generate property tests simultaneously. The agent often produces better properties when thinking about the code and tests together.
Be realistic: this approach works well for utility functions and algorithms but requires very different strategies for API handlers, database interactions, and UI logic.

Code Example

snippet

# Property-based testing example (using Hypothesis)
from hypothesis import given, strategies as st

@given(n=st.integers(min_value=1).map(lambda n: n * 3 * 5))
def test_returns_fizzbuzz_for_multiples_of_3_and_5(n: int) -> None:
    assert fizzbuzz(n) == "FizzBuzz"

# Mutation testing example - if there's a side effect, the mutant survives
# Even if we change print(f"DEBUG n={n}") to print(None) in the code below,
# test_doubles_input still passes → mutant survives = problem exists
def double(n: int):
    print(f"DEBUG n={n}")  # the test fails to catch this side effect
    return n * 2

def test_doubles_input():
    assert double(3) == 6

# Fix: remove the print or include the output in the test

Terminology

Property-based TestingA testing technique that automatically generates many inputs and verifies that specified invariants (properties) always hold, rather than testing specific examples.

Mutation TestingA technique that deliberately introduces small bugs (mutations) into code and checks whether the test suite detects them, measuring test coverage quality.

InvariantA condition that must always be true — e.g., 'sort output length equals input length' or 'all values are non-negative.'