The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents | AI Paper Digest

TL;DR Highlight

A single boot check fixed launch failures; a full shell cost 2.35x the tokens.

Who Should Read

An agent developer deciding how many verification tools (linter, shell, screenshots) to give a coding agent given token costs.

Core Mechanics

Blind builds fail to launch about 1 in 7
- Survival with no tools was 86.5% (166 of 192).
- Most failures came from files corrupted during edits or seed scripts that crash on first boot.
One boot probe is the best value
- A tool that only reports whether the server started lifted survival to 99.5%.
- Median tokens were 214k, below even the no-tools condition at 262k.
A shell buys depth at 2.35x cost
- Hard probes like race conditions and restart persistence needed actual execution.
- Still, 10 of the 12 functional points came from the boot probe alone.
The linter did nothing measurable
- Linter plus type checker was statistically indistinguishable from no tools.
- In one run the lint call failed because it was never wired up, and the model finished anyway.
Screenshots help only where faults are visible

Evidence

Shell vs no tools: functional +12.26
- On the 3 API-probed apps, execution − no_verification has 95% CI [+7.04, +17.47] and Holm p=0.0006.
Boot probe vs no tools: survival +0.13
- Across all 7 apps, 95% CI [+0.10, +0.16], Holm p=0.0006.
- Gemini gained 50 points of survival and gpt-5.5 gained 20.
Median token costs
- no_verification 262k, boot_check 214k, execution 615k, visual 674k, delayed_verification 1,199k.
- Boot check scored 91.7 versus the shell's 94.2 functional at about 35% of the cost.
Sight effect not significant after correction
- On the two visual tasks, visual − execution was +6.88 human score, 95% CI [+0.83, +13.41], Holm p=0.0826.
- On the log explorer negative control it was −2.17.

How to Apply

Add a boot-check tool to your agent first
- Today, add one tool that installs dependencies, starts the server, and returns whether the port is listening.
- Compare launch success against a no-tool run on the same task.
Match verification tools to failure types
- Give a screenshot tool for layout-heavy work.
- Give a shell for concurrency or performance work and have the agent check with load or racing requests.
Audit linter wiring and call logs
- Verify the lint script is actually configured.
- Add a rule that the agent must re-run checks before finishing if an earlier call errored.
Set verification budget per model
- Give models that already build well only the boot probe.
- Open the shell only for failure-prone models to save tokens.

Try It

copy & run

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: boot check + verification habit)

너는 소프트웨어 엔지니어다. 아래 과제를 구현하라.
과제: {{과제 설명}}

규칙:
1. 모든 파일을 쓴 뒤 종료하기 전에 반드시 의존성을 설치하고 서버를 실행해 포트가 리슨되는지 확인하라.
2. 서버가 안 뜨면 에러 로그를 읽고 고친 뒤 다시 실행하라.
3. 서버가 뜬 뒤 핵심 API를 curl로 한 번씩 호출해 응답을 확인하라.
4. 마지막에 확인한 항목과 확인하지 못한 항목(화면 렌더링, 클릭 동작 등)을 구분해 보고하라.

Next: run the same task again without these rules and compare launch success and token usage.

Terminology

verification surfaceThe set of tools an agent can use to check its own work, like handing it a toolkit of inspection gear.

boot probeA tool that installs dependencies, starts the server, and reports only whether the port opens. Like just turning the car key.

intention-to-treatGrading each run by its assigned condition even if the tool went unused, like counting a patient who never took the pill in the drug group.

Holm correctionA statistical adjustment that makes the bar stricter when testing several hypotheses at once, to filter out chance findings.

brownfieldWork that modifies existing code rather than starting from scratch.

Related Papers

Related Resources

Original Abstract (Expand)

Modern artificial-intelligence coding agents can be equipped with tools for checking their own work e.g. a linter, a boot probe, a shell, a screenshot tool. We call this set the agent's verification surface. This study asks whether increasing only that surface, with everything else held fixed, produces a matching growth in the quality of the software the agent ships. We built a minimal coding agent whose tool list is the single controlled variable and used it to implement 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric, and automatic probes stress-tested the API-observable behaviors. Verification's cheapest benefit arrives first, which is to make sure that the application comes up. Without any tools, about one build in seven fails to launch at all and a single boot probe removes nearly all of these failures at roughly 35 percent of a full shell's token cost, while the full shell multiplies the no-tools cost by 2.35. Screenshots help most where mistakes are visible (e.g. element placement, interaction), though even there the gain over a shell is modest and does not survive correction for multiple statistical comparisons. In cases where failures can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing. A verification tool improves the output artifact only where its reach covers the way the application actually fails.