The Silent Cost of Conversation History in LLM Reasoning
TL;DR Highlight
Keeping full conversation history dropped accuracy from 0.6 to 0.1 within 20 turns; a verified state summary prevented it.
Who Should Read
A backend developer building a multi-turn assistant that tracks budgets or inventory and pastes the whole chat log into every prompt.
Core Mechanics
- Dropping the model's own replies helped
- On 48 short math tasks, keeping the model's earlier replies in context lowered final success.
- GPT-4o-mini went from 0.896 to 0.688 and Mistral-7B from 0.354 to 0.063.
- 100-turn state tracking collapsed in 20 turns
- On CDR-Stream (a 100-turn state-tracking benchmark), accuracy with full history fell from about 0.6 in turns 1–5 to about 0.1 after turn 20.
- All three models showed nearly identical curves, in every scenario.
- Keep state outside, send a summary
- Sending only a solver-verified current state plus the new question scored 0.52–0.59 higher than full history.
- The prompt at turn 100 shrank from tens of thousands of tokens to a few hundred.
- Quality metrics did not warn
- Consistency and coherence fell far less than accuracy, and stability actually rose.
- A model that believes a wrong state keeps reasoning consistently from it.
- Drift detectors lacked power on one stream
- A single 100-turn stream needed an accuracy drop of about 0.5 to reach 80% detection.
- Pooling several dialogues lowered the detectable drop.
- Limits the author states
- Tasks were arithmetic and state tracking, and the models were not the strongest available.
- Study 2 did not isolate the effect of the assistant's replies alone.
Evidence
- Study 1 final success drop
- Over 48 tasks, GPT-4o-mini scored 0.688 with history vs 0.896 with a fresh prefix (no assistant replies), a 0.208 gap (Holm p=0.026).
- Mistral-7B scored 0.063 vs 0.354, a 0.292 gap (Holm p=0.004).
- Llama-4 Scout moved the same direction by 0.104 but not significantly (Holm p=0.125).
- Study 2 accuracy by turn range (history kept)
- GPT-4o-mini scored 0.61 in turns 1–5, 0.25 in turns 6–20, and 0.09 in turns 21–100.
- In the stationary easy scenario (S0-E), turns 21–100 scored 0.10–0.14 with history vs 0.82–0.86 with the state summary.
- Accuracy gap vs state summary (turns 1–100)
- The gap was 0.545 for Llama-3 70B and 0.588 for Gemma 3 12B (both Holm p<0.001).
- GPT-4o-mini showed 0.521 (95% CI [0.476, 0.566]).
- Quality metric trends (GPT-4o-mini, history kept)
- Accuracy fell from 0.61 (turns 1–5) to 0.08 (turns 81–100).
- Stability rose from 0.94 to 0.98 and coherence dipped slightly from 0.54 to 0.47.
How to Apply
- Re-measure state questions in your own logs today
- Pick 20 questions about current state and score accuracy in early turns vs after turn 20.
- Re-run the same questions with only a current-state summary and no history, then compare.
- Store state externally and send it each request
- Keep ledger, inventory, or schedule state in a DB or JSON updated by code.
- Put only the current state and the new question into each prompt.
- Add a verification step for model-written summaries
- Check any model-written state summary with rule-based computation or a solver before passing it to the next turn.
- The paper expects model-written summaries to do worse than its solver-verified one.
- Monitor by pooling sessions
- Do not use a single conversation's consistency or stability dashboard as an accuracy proxy.
- Aggregate across sessions or compare against history-free re-queries.
Try It
# Simplified version of the paper's method: paste into a fresh Claude/ChatGPT chat (state-summary vs full-history comparison)
[Role] You are a warehouse inventory ledger assistant. Answer using only the 'Current state' below. No earlier conversation is provided.
[Current state]
{{current ledger state (e.g., Warehouse A part X 120, Warehouse B part X 40)}}
[Question for this turn]
{{question (e.g., total stock of part X?)}}
Write the answer as 'FINAL_ANSWER: <number>' and give the calculation in one line.
Next step: ask the same question with {{20 turns of full chat log}} attached, and compare both answers to the correct one.Terminology
Related Papers
LLM assisted reverse engineering
Pointing an agent at binaries with skills and MD notes made years-long hacks feasible
Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code
Injecting per-turn complexity reports into coding agents cut code erosion without changing correctness
Why is Gemini 4 Carbon taking so long?
Carbon took nearly an hour on bug fixes that Astra and Opus finished in minutes, mostly from extra unrequested work.
How the Claude Code team thinks about verification
E2E, property and differential tests plus verification rules let Claude run in parallel.
Running coding agents without handing them my credentials
Credentials stay on the laptop; agents run commands there via Touch ID approval
Windows 11 Is Collecting Data on Which Coding Agents and Developer Tools You Use
Reverse engineering shows Windows 11 logs which coding agents you use and reports it as optional diagnostics
Anthropic upgrades Claude Code Projects to run parallel AI workers
Claude Code Projects now has a coordinator that splits work into parallel threads and opens PRs.
Show HN: AgentRun – open-source, Self-improving Managed Agents
Agents save notes and skills after each run, cutting code review cost to 18 cents
Related Resources
Original Abstract (Expand)
Large language models (LLMs) in multi-turn information systems accumulate their own earlier replies and a growing task state. We asked whether this conversation history lowers accuracy, and whether process-level quality metrics warn before it does. Each analysis was fixed before its outcomes were computed. In Study 1, on 48 sharded math tasks, retaining its replies lowered final success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B (Holm-adjusted p = 0.026 and 0.004); Llama-4 Scout showed the same direction, not significant. Study 2 introduces CDR-Stream, 100-turn state-tracking dialogues with an exact answer per turn and per-model difficulty calibration. With the full history, accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, whereas a solver-verified state summary kept it 0.52-0.59 higher (Holm-adjusted p less than 0.001). Consistency and logical coherence fell far less than accuracy, and stability rose. None of these metrics, a paired readiness measure or calibrated single-stream drift detectors gave robust, model-general early warning, although in Study 2 the repeat-based metrics were complete only for GPT-4o-mini. In the monitors we tested, a single 100-turn stream had power only for very large drops. Systems built on LLMs should keep task state outside the transcript and monitor across sessions.