Chained Recursive Language Models for Multi-Iteration Reasoning | AI Paper Digest

TL;DR Highlight

Chaining fresh calls of one model with plain-text handoffs raised long-context accuracy.

Who Should Read

An LLM agent developer whose long-document counting, aggregation, or ordering answers keep coming out wrong

Core Mechanics

Solve in stages, not in one shot
- The same model is called several times as fresh roots, each getting only the original input plus a summary and notes.
- Prior conversation history is not passed by default.
Only three plain-text items are handed off
- The handoff has SUMMARY, BLACKBOARD, and NEXT in plain text.
- No JSON, XML, or YAML parsing is used.
Intermediate work goes into files (artifacts)
- For a counting task, the first root writes a ledger with one row per candidate event.
- The next root fixes and audits the ledger before the final answer.
A fresh root re-reads earlier mistakes
- In a single trajectory the model tends to keep trusting conclusions it already wrote.
- A fresh root treats them as external files and checks them again.
Harder tasks gain more

Evidence

Benchmark accuracy (GPT-5-mini, vs regular LLM)
- RULER 87% to 92%, BABILong 44% to 59%.
- LongBench v2 41% to 52%, OOLONG-real 14% to 38%.
Average accuracy gain
- +13.75 percentage points absolute across the four benchmarks.
Average cost (vs regular LLM)
- Cost per task $0.125 to $0.315.
- Root calls 1.0 to 2.6, input 48.8k to 98.8k tokens, output 5.8k to 21.8k tokens.
OOLONG-real cost
- Cost per task $0.14 to $0.44, with 3.4 root calls and 2.4 handoffs.

How to Apply

Try a two-step prompt on a long document
- Step 1 only builds a candidate ledger, and step 2 verifies and aggregates it in a fresh chat with the original text.
- Paste the codeExample as is to start.
Fix your handoff to SUMMARY/BLACKBOARD/NEXT plain text
- Shrink the state passed between agent steps to these three fields.
- Paste them into the next call's prompt without a JSON parser.
Turn on chaining only for hard tasks
- Send retrieval-style questions to a single call.
- Route counting, aggregation, and multi-evidence questions to the chain.
Log max roots and cost together
- Set a chain-length cap R and force the last root to give its best supported answer.
- Compare per-task tokens and cost against a single call to judge the gain.

Try It

copy & run

# Paste into a new Claude/ChatGPT chat (simplified version of the paper's method: a manual 2-root chain)

[Step 1: Root 0]
You are the first root for a long-document task. From the document below, extract candidate items needed for the question, one per line, as a tab-separated ledger (ledger.tsv). Do not use JSON, markdown tables, or code fences. Do not give an answer yet.
End with a handoff in this format.
HANDOFF:
SUMMARY: what you did and what you currently believe
BLACKBOARD: verified facts, assumptions, open questions
NEXT: what the next root should check

Document: {{document}}
Question: {{question}}

[Step 2: Root 1 in a new chat]
You are a fresh root for the same task. There is no prior conversation. You only have the original text, the ledger, and the handoff below. Keep the ledger's structure and fix definition errors, duplicates, and omissions. State what you changed, recompute from the fixed ledger, and end with FINAL: <answer>.

Document: {{document}}
Question: {{question}}
Ledger: {{step 1 ledger}}
Handoff: {{step 1 handoff}}

Next, try this: compare the result with the answer from solving the same question in one shot.

Terminology

rootOne call in the chain. It gets the original input but starts fresh with no prior conversation history.

blackboardA plain-text notepad left for the next root, holding the current best answer, verified facts, and open questions.

artifactA plain-text work file a root saves to disk, such as a candidate ledger or a verification checklist.

context rotThe drop in model performance as the context grows long and the model loses track of the work's structure.

RLM (Recursive Language Model)A method that treats a long input as an external environment and lets the model recursively call itself on the relevant parts.

Related Papers

Original Abstract (Expand)

Long context reasoning in large language models (LLMs) is usually constrained by the fact that a single inference trajectory has to simultaneously explore the context, store intermediate state, verify evidence, and produce the final answer. This becomes particularly difficult in tasks that require extraction, counting, ordering, or multi-hop reasoning, where an early mistake can propagate until the final response. In this work, we propose Chained Recursive Language Models (Chained RLM), an inference-time architecture, in which the same underlying model is called repeatedly as a sequence of fresh reasoning roots. Each root receives the original problem and context, but does not inherit the full conversational history. Instead, it receives a compact plain-text summary, a plain-text blackboard, and some durable task-specific artifacts written by predecessor roots. The motivation is to manage the context by chopping into partial tasks rather than one large inference response; in each staged computation, intermediate artifacts can be inspected, corrected, and extended by a later fresh inference by the same model. We describe the system model, handoff mechanism, artifact workspace, and evaluation protocol for this system. We study when fresh-context artifact continuation gives a measurable gain in accuracy over direct LLM answering even with recursive tool-calling.