Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy | AI Paper Digest

TL;DR Highlight

Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

Who Should Read

Backend or agent developers whose harness saves long tool outputs to files for the agent to read instead of inlining them in context.

Core Mechanics

File delivery hurt some models, not others
- Giving the PHREEQC (geochemistry simulator) report as a table of contents (TOC) cost GPT-5.1 11.3 points and GPT-5.2 7.9 points versus inline.
- Claude Opus 4.6, Sonnet 4.6, and GPT-5.4 changed by -0.6 to +0.2 points, indistinguishable from zero.
The computation was right but the answer wasn't
- GPT-5.1 and GPT-5.2 produced the keyed value about equally often under every interface.
- Under file delivery they skipped the report and answered from their own SELECTED_OUTPUT files.
Their own files had more digits, so rounding broke
- The report prints values at the options' precision, while the agent's own files print more digits.
- 2.75065 was truncated to 2.750 instead of rounded to 2.751, picking a neighboring option (the paper calls this evidence substitution).
A net effect near zero can hide a loss
- GPT-5.4 lost about 10 points to the same substitution.
- Its simulations produced the keyed value more often under retrieval, so the net change looked like zero.

Evidence

Accuracy drop of TOC vs inline
- On 158 items: GPT-5.1 -11.3 points ([-18.0, -4.9], Holm p=0.005), GPT-5.2 -7.9 points ([-14.9, -0.9], Holm p=0.128).
- Claude Opus 4.6 -0.6, Sonnet 4.6 -0.5, GPT-5.4 +0.2 points, all with intervals including zero.
Loss persists with the simulation held fixed
- GPT-5.2 on 131 frozen checkpoints: 71.2% inline vs 61.8% TOC, a -9.4 point difference ([-14.5, -4.6]).
Report skipping and the rounding trap
- Under TOC, no part of the report reached the agent in 60.3% (GPT-5.2), 53.0% (GPT-5.1), and 10.5% (Sonnet) of trajectories.
- The truncated neighbor was chosen in 26/35 (GPT-5.2), 19/27 (GPT-5.1), and 1/35 (Sonnet) trap cases.
Effect of interventions
- The report instruction raised accuracy over all TOC runs by +6.3 (GPT-5.1), +7.9 (GPT-5.2), and +7.8 points (GPT-5.4).
- The rounding-only instruction (TOC+Round) gave just +0.6 and +1.2 points, with closure staying around 82%.

How to Apply

Name the authoritative output file in the tool description
- In the description and every result message of the tool that saves long output to files, tell the agent to take values from that file and not from files it wrote itself.
- The paper's instruction states the report prints values at the options' precision and warns that unrounded output can match a neighboring option.
Inline outputs that fit in context
- If the output is under your budget (the paper used 100,000 characters), put it directly in the context and use a TOC or search only for longer ones.
- The paper's Hybrid setup did this and gained +4.6 (GPT-5.2) and +8.0 points (GPT-5.1) over the TOC control run.
Break down losses by stage in your logs
- Don't look only at final accuracy; count whether the value was produced, whether the agent saw it, and whether it answered correctly given it saw it.
- Run both inline and file delivery whenever you switch models and compare.

Try It

copy & run

# Paste into a Claude/ChatGPT system prompt or tool description (simplified version of the paper's TOC+Report instruction)

IMPORTANT: The authoritative values are in {{authoritative output file}}, printed at the precision the answer options use. Before answering, read the section of {{authoritative output file}} that contains the quantity you need and take the value exactly as printed there. Do not answer from {{kind of agent-created output files}} files: they print unrounded values that can match a neighboring option.

# Next: run the same 20 questions with and without this instruction and compare accuracy and how often the authoritative file was opened

Terminology

PHREEQCA geochemistry simulator from the USGS. It is deterministic, so every printed value can be traced and verified.

TOC deliveryInstead of pasting a long report, give only a table of contents of its sections. The agent then reads just the line ranges it needs.

Evidence substitutionAnswering from output the agent defined itself instead of the tool's canonical report. Like taking an exam from your own notes instead of the official reference.

ClosureAmong cases where the agent saw the keyed value, the share it answered correctly. Misrounding a value it saw shows up here.

Shapley decompositionA way to split an accuracy change among build, exposure, closure, and unsupported stages. It does not estimate independent causal effects.

Related Papers

Original Abstract (Expand)

LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced. Agents use it to answer 158 execution-verified multiple-choice items. When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running all 11,007 logged simulator calls shows that both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not. Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.