You don't need long context

Who Should Read

LLM app developers whose long agent loops or long-document QA are burning too many tokens and too much latency

Core Mechanics

• Later tokens cost more
  • Cutting the context window from 256K to 16K lowers per-token cost by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.
  • Retail prices are flat regardless of context length, so users never see this gap.
• Agents don't need to carry full history
  • When context hit a limit, the agent summarized (compaction) and continued in a fresh conversation (EoCC).
  • Both Claude Opus 4.5 and GPT 5.6 Sol kept accuracy nearly intact in most settings.
• Put long docs in files and grep them
  • Long inputs were stored as files and the agent read only needed parts with grep and sed.
  • This turned long-input tasks into agent tasks, so context didn't grow with document length.
• Compacting after every action still works
  • Each action got a fresh conversation carrying only a compact semantic state (SAC).
  • Median peak context was 7–9K tokens and accuracy mostly held.
• It doesn't work for every model
  • Open-weight Qwen 3 Coder Next collapsed on SWE-bench Verified from 53.0% to 3.6% at 16K/12K.
  • Verbose models had too little room to make progress between compactions.
• Compaction is lossy
  • Tasks needing every record, like a coin toss log, will fail after compaction.
  • The fix is to save the full log to a file and read it selectively when needed.

Evidence

• Long-horizon accuracy barely moved while costs fell a lot
  • Opus 4.5 changed -3.7 to +5.6pp with 2.2–8.8x cost savings versus full history.
  • GPT 5.6 Sol changed -3.3 to +4.4pp with 2.4–5.0x savings.
• Selective reading beat full input on AA-LCR
  • At 16K/12K, Opus 4.5 accuracy rose from 65.0% to 84.0%.
  • GPT 5.6 Sol rose from 82.0% to 83.0%.
• RULER and BABILong jumped
  • RULER recall rose from 62.5% to 95.7% for Opus 4.5 and from 65.5% to 96.4% for GPT 5.6 Sol.
  • BABILong accuracy rose from 67.1% to 92.0% for Opus 4.5 and from 70.7% to 94.0% for GPT 5.6 Sol.
• SAC held or improved accuracy on Claude
  • Opus 4.5 on Terminal-Bench went from 47.2% (mini baseline) to 51.8% with SAC.
  • GPT 5.6 Sol dropped slightly: SWE-bench Verified 84.4% to 82.4%, Terminal-Bench 87.6% to 84.3%.

How to Apply

• Add a compaction prompt to your agent
  • When the conversation nears 16K–48K tokens, insert one handoff-recap request into your current agent loop.
  • Continue in a fresh conversation containing only the system prompt, original task, and the recap.
• Keep long docs in files and let the agent grep
  • Replace pasting whole documents into prompts in your RAG or doc-QA flow with saving them to files.
  • Give the agent bash tools like grep, sed, and head, and cap each read at about 6000 characters.
• Measure accuracy and cost per compaction threshold
  • Compare full history against 48K, 32K, and 16K thresholds on your own eval set.
  • Log accuracy and total tokens together to check whether retries eat the savings.
• Save must-keep information to files
  • Store all tool results in a log file the agent can read on demand.
  • Have the recap include file paths and the commands to read them.

Try It

# Prompt to paste into Claude Code or an agent loop when context gets long (simplified version of the paper's method)

Stop work. Write only a concise handoff recap; do not call tools.
The conversation will reset, and the system prompt, task, and recap will be provided again.
Files and environment state remain available: {{persistent files and environment}}

Preserve what your next stretch needs:
1. Current progress and best answer/result, separating verified facts from assumptions.
2. Relevant evidence with exact values and source locations; changes already made, with paths and verification status.
3. What was solved, what worked, what remains unresolved, what failed, and useful commands already tried.
4. Remaining plan - outstanding subgoals in execution order; the immediate next command or edit; and the condition for completion.
If important output was truncated, name a narrow command to retrieve the missing part. Do not invent observations or repeat the full task.

Next step: put this recap into the first message of a new conversation along with the original task and check that work continues.

Terminology

Related Papers

• Progress on provable inference for large language models

  Proving 400B model outputs with zkSNARKs works, at 70-95x the GPU time of plain vLLM

• Tell HN: Gemini trains on chat history, even for subscribers (personal accounts)

  Gemini personal accounts may train on and human-review chats, even for paying subscribers

• The AI cost of JavaScript frameworks

  Building the same app with 12 JS frameworks via agents, 5 runs each, showed up to ~2x cost differences.

• Claude Code regularly sends each background agent’s transcript to its model

  Disabling Claude Code's 30-second agent status summaries removed 19.7% of API requests in one team's telemetry

• Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

  A KV pool smaller than the context window leaked other customers' data into 155 long-prompt responses.

• Microsoft Decision-1: 9B Qwen Model Launch Explained

  Microsoft's 9B Qwen-based model returns only per-option probabilities for routing and classification.

• Cerebras Deploys Trillion-Parameter AI Models

  Cerebras fits Kimi K3 (3T) across 59 wafers in a pipeline, serving 1000-2000+ tokens per second

• A practical guide to AI image generation, from diffusion to agents

  Testing image APIs showed you pay per canvas, and camera-style prompts beat describing the view