When the harness decides success: state that cannot be re-observed
TL;DR Highlight
Truncating context costs nothing if facts can be re-looked-up, but drops success to chance when they can't
Who Should Read
An agent developer deciding whether to truncate or summarize old turns in long-running agent loops
Core Mechanics
- Truncation is fine if the fact can be looked up again
- With a truncated window, gpt-6-luna scored 100% when the fact was recoverable.
- It simply looked the fact up again.
- A fact that can't be re-seen is gone for good
- In the same window, success on non-recoverable facts was 7.5%, near the 10% chance level.
- The gap between the two penalties is 92.5 points.
- Models behave differently after losing a fact
- claude-haiku-5-5 chose no answer in all 400 non-recoverable far episodes.
- gpt-6-luna picked an option in 387 of 400 and was right in 31.
- A note-taking line plus summaries closed most of the gap
- On ARC-AGI-3, summarizing instead of dropping while keeping the note-taking instruction recovered 86% of a 63-point gap.
- This happened without replaying the model's reasoning.
- Truncation also raises cost
- The window used 0.84-0.90x the tokens but cost 2.9-4.0x as much as full history.
- Dropping turns breaks prompt caching.
- The ARC results are single runs, so don't over-read them
- There was one counted run per configuration and game.
- The replay share falls from 57% to 35% if a different FT09 S2 run is counted.
Evidence
- PersistGym-lite window penalty gap of 92.5 points
- gpt-6-luna with a window (W-0): 7.5% (15/200) not recoverable vs 100% recoverable; full history 200/200 in both.
- ΔH1 = +92.5 points (90% CI [89.5, 95.5]).
- Replication on a second provider's model
- claude-haiku-5-5 with a window: 0/200 not recoverable, 200/200 recoverable.
- ΔH1 = +101.0 points (90% CI [100.0, 102.5]).
- Caching and cost numbers
- 94% of full-history input tokens were served from cache vs 48% for the window.
- Window dollar cost was 2.9-4.0x full history on gpt-6-luna and 3.3-3.7x on claude-haiku-5-5.
- ARC-AGI-3 gap decomposition (Gemini 3.8 Flash, 4 games)
- Mean RHAE: S0 (Standard) 9.24, S1 (summary + notes) 63.08, S2 (summary, no notes) 36.32, S4 (Adapter) 71.83.
- Of gap G=62.59: σloss 0.860, σnotes −0.427, σreplay 0.567.
How to Apply
- Add one note-taking line to the system prompt
- For agents that truncate old turns, add "Include any context you want to carry forward in your reply, along with the action you want to take."
- Run the same tasks with and without it and compare success rates.
- List state the agent can't re-observe
- Replace dropping with summarization
- When context fills, replace old turns with a model-written summary instead of dropping them.
- Give the summary request a large output budget (the paper raised 8,192 to 65,536 after repeated empty replies).
- Check cache hit rate when truncating
- Measure cache hit rate and dollar cost before and after switching to a window.
- Fewer tokens can still mean higher cost.
Try It
# Paste into Claude or ChatGPT (simplified version of the paper's method: audit state that can't be re-observed)
I'm building an LLM agent. It drops old turns when the context fills.
Below are the agent's workflow and tool list.
{{workflow and tool descriptions}}
Instructions:
1. List all information that is shown only once or can't be re-fetched by any tool (computed results, one-time outputs, etc.).
2. For each item, classify it as 'recoverable' or 'permanently lost' if old turns are truncated.
3. For each permanently lost item, suggest one of: save to file/notes, restate in every reply, include in the summary.
Next: add "Include any context you want to carry forward in your reply, along with the action you want to take." to the system prompt and compare success rates.Terminology
Related Papers
Here be monsters: What do you still need to know when agents do the work?
Owning agent-written code depends on fundamentals, breadth, and diving deep just in time.
AI promised less work. I've never worked more
Running agents in parallel brought my GitHub activity back to full-time engineer levels, and I'm exhausted.
Show HN: Corgi, a helpful {agent, terminal} multiplexer
A tmux-like terminal tool that shows the status of all your AI agents at a glance
Ask HN: If the models are so good, why doesn't OpenAI just start 100 companies?
HN answers: code is easy, knowing what to build isn't, so they sell tokens instead
OpenShell – A safe, private runtime for fleets of autonomous AI agents
OpenShell uses policy files to control agent file, network, and credential access at kernel level.
Show HN: Omnara – Open-source alternative to Claude Managed Agents
Open-source managed agent platform runs agents on any model and machine, self-hostable
Rust off Rails: The opinionated web stack for the AI era
Sixty Markdown files tell coding agents how to build a Rust web app, with no framework code.
LayerFS: The filesystem for AI coding agents
Mount an ephemeral workspace per agent, commit only the changes
Related Resources
Original Abstract (Expand)
Agent harnesses decide what a language model sees across turns: they drop old turns, summarize them, or replay the model's own earlier reasoning. We ask when these choices change whether an agent succeeds, not only what it costs, and report three findings. First, in a pre-registered synthetic probe, truncating the context window costs a low-cost reasoning model nothing in success when a needed fact can be looked up again (it looks the fact up instead), but drops its success to chance when the fact cannot be re-observed: the two penalties differ by 92.5 percentage points (90% CI [89.5, 95.5]). A pre-registered replication on one model from a second provider showed the same contrast. Once the fact was lost, that model chose no answer rather than guessing. Second, on ARC-AGI-3, where the benchmark's Standard harness trails its Provider Adapter harness by a wide margin, we decompose that gap one component at a time, for one model on four games. Summarizing old turns instead of dropping them, while keeping the Standard harness's invitation to take notes, recovers 86% of a 63-point gap without replaying the model's reasoning. The same harness without the invitation recovers 43%, and adding the provider's native replay of the model's reasoning recovers the remaining 57%. These are point estimates from one counted run per configuration and game, and the replay share depends on a single run. Third, in the benchmark's published runs and in ours, the Adapter's advantage in the probability of clearing a level it has reached is larger at deep levels than at early ones. This largely restates where runs stopped and does not separate a growing advantage from a constant one.