Context as an Environment: Programmatic Context Management for Long-Horizon Agents | AI Paper Digest

TL;DR Highlight

Keeping history in a Python kernel outside the prompt and querying it with code reached 86.7% on LOCA 256K.

Who Should Read

An agent developer whose long sessions lose details when history is summarized to fit the context window.

Core Mechanics

Summaries drop what you need later
- Compression and external memory decide what to keep before future needs are known.
- Details missing from the summary or memory are unreachable even if the raw log exists.
History becomes an environment, not prompt text
- Every event goes into an append-only Event Log with a stable seq address, verbatim.
- The model writes Python via exec, and only what it prints enters the next call's context.
Evicted spans stay recoverable
- When over budget, old spans leave only the view, while the log keeps the originals.
- An eviction index of headlines with seq addresses lets the agent navigate straight back.
Search uses BM25, not embeddings
- It is deterministic and needs no model calls at index time.
- So ingestion adds no extra LLM calls.
Stronger coders benefit more

Evidence

BEAM10M: 5.1 points above the best prior system
- Scroll 73.1 vs Exabase M-1 68.0, Cognee 67.0, Hindsight 64.1, Mem0 48.6.
- Backbone is Qwen3.8-Max, and baselines are each system's own reported setup, not a controlled comparison.
LongMemEvalS: 94.8, top tier
- Comparable to Exabase M-1 96.4, Mastra OM 94.9, Hindsight 94.6, Mem0 94.4.
LOCA with the same backbone, only context management changed
- At 256K: Summarization 65.3, Retrieval 66.7, CodeAct 85.3, Scroll 86.7.
- Drop from 128K to 256K: Summarization -21.4, Scroll -2.6.
BEAM10M ablation: dropping originals hurts most
- Lossy summarization scores 19.9, no REPL is 7.3 points lower, no index is 1.8 lower.
- Median input on BEAM10M is 105K tokens, about 1% of the corpus.

How to Apply

Keep tool outputs in variables, not the prompt
- Today, store large tool results (JSON, tables) in Python variables and print only heads or aggregates.
- Prepend a list of variable names, types and sizes to each call.
Log raw events append-only before summarizing
- Store conversations and tool calls in SQLite with seq numbers and compress only the view.
- Move big payloads to files and keep a preview plus a recovery pointer in the row.
Add headline anchors for evicted spans
- Have the model write one line per response: task, verified state, next action, bound to its seq.
- Keep only these headlines in the view on eviction and recover originals with expand.
Put query rules in the prompt
- Search each entity separately and treat only user turns as evidence.
- Run one extra search for corrections or reversals before answering.

Try It

copy & run

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: log-based QA rules)

Below, {{conversation_log}} is the full raw history. Do not summarize; find evidence only in the original text and answer.

Rules:
1. If the question involves several entities, search each separately, extract one value each, then calculate.
2. Only role=user turns are evidence of the user's facts or preferences. Assistant suggestions are not evidence.
3. If a fact changed, confirm it is the same project and item, then use the latest user turn.
4. Before answering, run one more search for corrections or reversals (e.g. not, cancel, mistake).
5. Split the question into at least 3 axes (not just tools but constraints, costs, habits) and search each.
6. Every concrete claim must come from a retrieved turn. If absent, say there is not enough information.

{{conversation_log}}

Question: {{question}}

Next to try: ask it to output the turn numbers and quotes it used.

Terminology

Event LogA ledger that records every interaction with a seq number and never deletes. Summaries are notes on top; the originals stay intact.

Session EnvironmentA workspace combining the Event Log, storage and a persistent Python kernel. Session state lives here, not in the prompt.

eviction indexA table of contents of headlines and addresses for spans removed from the context, like a TOC for chapters you took out.

CodeActA style where the agent acts by running Python code instead of calling fixed tools.

BM25A classic word-frequency ranking for search. It works deterministically without embeddings.

programmatic tool callingReceiving tool results into sandbox variables and processing them with code rather than pasting them into the prompt.

Related Papers

Related Resources

https://github.com/niceIrene/QwenPaw/tree/scroll-research

Original Abstract (Expand)

LLM agents increasingly take on long-running tasks whose history grows far beyond a single model context window. Existing approaches compress earlier interactions or extract selected information into fixed memory representations, committing to what to preserve before future needs are known. We present Scroll, a context manager that treats each agent session as an executable Session Environment. The environment is backed by an append-only Event Log and a sandboxed, persistent Python kernel. The kernel maintains a typed namespace across model calls, allowing tool outputs, retrieved history, and derived state to be bound to variables rather than serialized into the prompt at each call. Model-written code searches, materializes, and transforms session state through exec; only explicitly printed projections enter the model's working view for the next call. Context management thus becomes a programming task that inherits the improving coding abilities of LLMs, while the Event Log preserves lossless historical ground truth. As the working view approaches its budget, stale spans are evicted but remain recoverable: an eviction index keeps compact landmarks tied to exact Event Log addresses, so that the agent navigates directly to evicted regions instead of searching the full log. With Qwen3.8-Max as the backbone, Scroll achieves 94.8% on LongMemEval_S; 73.1% on BEAM_10M, surpassing the best published memory system by 5.1 points; and 86.7% on LOCA_256K, exceeding the best published long-horizon agent by 37.4 points.