Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents | AI Paper Digest

TL;DR Highlight

Pruning source-less entries from compaction summaries raised Pass3 by 2.1–8.9 points

Who Should Read

An agent developer whose long-running agents go off track after context compaction summaries

Core Mechanics

Compaction summaries mix in compressor guesses
- Summary prompts ask for decisions and next steps, so environment facts and the compressor's plans end up in one summary.
- After compaction the agent can't tell them apart and treats everything as fact.
67.7% of TODOs and 86.1% of user asks have no source
- Across 26,881 OpenClaw summary entries, 67.7% of Open TODOs and 86.1% of Pending user asks had no source.
- Only 3.4% of Exact identifiers entries were unsupported.
29 of 35 failures trace to a summary entry
- Of the 29 failing actions, 22 carried out unsupported entries and 3 carried out entries contradicting an environment output.
- One example: the user asked for the movie titles the friend requested, and the summary said to send the whole list.
SJC asks for a source per entry and drops the rest
- It splits the summary into entries, retrieves evidence with BM25, and gets scores for all entries from the Jev judge in one request.
- Entries scoring below 0.5 are removed whole, and kept entries stay verbatim.

Evidence

Pass3 improved in all four settings
- Versus Prompting on AppWorld: GPT-5.6 Luna +8.9 points, MiniMax-M3 +7.1 points.
- On OfficeBench: +5.3 and +2.1 points respectively.
Bigger gains on harder tasks
- AppWorld Hard with GPT-5.6 Luna: Pass3 went from 39.7% to 52.4%.
- OfficeBench 3-APP with GPT-5.6 Luna: Pass3 41.9% to 54.8%, accuracy 59.1% to 64.5%.
Overhead is small
- Judge cost is $1.19–$1.52 per 1,000 tasks.
- 13–21% of entries were pruned and summary length dropped 13.7–26.7%.
Jev is steadier than self-judging
- Over 5 repeated requests, Jev changed its answer on 2.1% of entries and the compressor self-judge on 8.6%.
- The self-judge fell below Prompting in Pass3 in 3 of 4 settings.

How to Apply

Count unsupported summary entries in failed logs
- Take a few failed agent runs from today and compare the post-compaction summary with the preceding tool outputs.
- Find TODOs or decisions absent from tool outputs and mark whether they caused the failing action.
Split plan sections from facts in the summary prompt
- Make the summary use separate sections for facts (IDs, values, results) and plans (TODOs, decisions).
- Tell the agent in its prompt that the plan section is reference only, not instructions.
Add a source check right after compaction
- Split the summary into entries, attach the top 3 BM25 evidence passages, and ask a judge model in one request.
- Drop entries scoring below 0.5 and use the pruned summary as the previous summary at the next compaction.
Keep the task instruction out of the summary
- Leave the original task instruction verbatim in the prompt instead of restating it in summaries.
- In multi-turn setups, earlier user messages must count as sources, which the paper did not evaluate.

Try It

copy & run

# Paste into Claude/ChatGPT (simplified version of the paper's method: per-entry source check on a compaction summary)

You are a summary auditor. For each entry (bullet line) in [Summary] below, decide whether it contains information that came from [Environment outputs] or [Previous summary] (a value, a name, a result, or how something in the environment behaves). If it only has plans, decisions, or readings of the task, mark it 'unsupported'. [Task instruction] does not count as a source.

[Task instruction]
{{task instruction}}

[Previous summary]
{{previous summary}}

[Environment outputs]
{{tool outputs}}

[Summary]
{{compaction summary}}

Output a table per entry: 'entry / supported or unsupported / one-line evidence quote'. Finally, output the summary with only the unsupported entries removed, verbatim.

Next step: run the same request 3 times and check whether any entries flip.

Terminology

Context compactionReplacing the older part of a long history with a summary when it gets too long. It's like using meeting minutes instead of the whole meeting.

Pass3The share of tasks solved in all 3 runs. It measures consistency rather than luck.

ProvenanceWhere a piece of information came from. It separates facts the environment reported from plans the model made up.

JevA model that doesn't generate text and only returns a calibrated probability for a yes/no question. Think of it as a grader that outputs just a score.

BM25A classic keyword-matching method for ranking text passages by relevance. Here it finds the evidence passages for each entry.

Related Papers

Related Resources

Original Abstract (Expand)

Long-horizon LLM agents rely on context compaction: when the interaction history outgrows the context window, a compressor replaces it with a natural-language summary. We show that these summaries contain more than a record of what happened. Compaction prompts ask for decisions and next steps, so each summary mixes facts established by the environment with reasoning written by the compressor, and the agent cannot tell the two apart afterwards. In an analysis of a widely used structured compressor, 83% of failed runs trace back to an action that carries out a specific summary entry, most of which no environment output supports. We propose Source-Judged Compaction (SJC), a training-free post-processing step that scores each summary entry for whether it has a source in the environment outputs or the previous summary, and removes the entries that do not. It changes neither the agent nor the compressor and adds one inexpensive judge request per compaction. The judge is Jev, a non-generative decision model that answers the source question with a calibrated probability instead of text. On AppWorld and OfficeBench with two agent models, SJC improves cross-run reliability over the underlying compressor by 2.1%-8.9% without increasing the agent's token usage.