An Empirical Study of Harness Design for Coding Agents

Who Should Read

A backend developer building a coding-agent loop who must decide on context management, planning, and tool setup.

Core Mechanics

• Context management matters most when the window is tight
  • The gap between managed tiers (T1–T4) and unmanaged T0 on SWE-Bench shrank from 35.7 points at 32k to 15.9, 5.5, and 2.7 at 64k, 96k, and 128k.
  • Most of the benefit comes from avoiding context-overflow failures.
• Elide first, summarize only if needed
  • T4 replaces stale tool outputs with stubs first and uses LLM summarization only when still over budget.
  • It had similar success to other managed tiers with the lowest cost.
• The recall tool for elided content is rarely used
  • T2 (elision plus recall) beat T1 (elision only) in 15 settings, lost in 14, and tied in 3.
  • Models almost never call recall_event, so there was no accuracy gain.
• Planning is a scaffold for weak models and a saver for strong ones
  • Weak models reach an edit more often with planning, while strong models skip redundant post-edit verification.
  • Strong models keep nearly the same accuracy.

Evidence

• Context management helped most at 32k
  • On SWE-Bench, the model-averaged managed-vs-T0 gap fell from 35.7 points at 32k to 2.7 at 128k.
  • T0 overflow failures fell from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench, while managed tiers had zero.
• T4 had the lowest cost in 7 of 8 panels
  • Across four models and two benchmarks, T4 had success comparable to T1–T3 with the lowest cost in seven panels.
  • Averaged over the eight pairs, T4 had the lowest mean cost per task at every window budget.
• Recall went unused in 56.3% of settings
  • Of 64 T2 and T4 settings, 36 (56.3%) never called recall_event.
  • Mean calls per task fell from 0.540 at 32k to 0.007 at 128k, and the mean T2-vs-T1 difference was -0.36 points.
• Planning effects differed by model size
  • Nemotron-3 30B gained 11.6 points on SWE-Bench and 4.5 on Terminal-Bench from planning, at higher cost.
  • On SWE-Bench, Nemotron-3 550B and Mistral-Medium-3.5 cut cost about 30% and 32% with success down 2.0 and 0.4 points.

How to Apply

• Add elision of stale tool outputs to your loop
  • When tokens exceed 60% of the window, replace old bulky tool outputs with a stub like "[tool output elided: N lines]".
  • Keep the system prompt, original task, and at least the last two turns verbatim.
• Summarize only above 85% of the window
  • If elision is not enough, summarize the oldest events with the same model.
  • You can skip the recall tool at first, since the paper found no accuracy loss without it.
• A/B test planning per model
  • Run the same tasks with an update_plan tool and first-turn plan request on and off.
  • Compare success rate, mean turns, share ending without an edit, and cost.
• Try a bash-only setup on shell-heavy tasks
  • If the model is strong with the shell, drop read, edit, and grep tools and compare against the full set.
  • Check first whether runs end early from emitting tool names that do not exist.

Try It

# Paste into Claude/ChatGPT — simplified version of the paper's method (T4-style history summarization)

You compact a coding agent's conversation. Below is the OLDEST part of the transcript. Respond with TEXT ONLY, do not call tools.
Summarize concisely under these headings (omit if empty):

## Goal
## Files touched
## Done
## Pending
## Errors & fixes
## Current state
## Next step
Keep exact file paths, function names, and error messages. If a previous summary is given, update it.

[Previous summary]
{{previous summary}}

[Oldest transcript]
{{transcript}}

Next to try: first replace long tool outputs with "[tool output elided: N lines]" stubs, and use this summary prompt only if the history is still too long, then compare cost.

Terminology

Related Papers

• Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

  Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

• Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents

  Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.

• Harness Engineering 101: The LLM is the brain, the harness is the body

  An agent is a loop managing a JSON array; 16 chapters build one from scratch

• Harness2 by Google Research – Recursive Agent Harnessing

  Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.

• Ousted Savannah police officer gave ICE agent husband access to Flock system

  A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired

• Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling

  Running Claude Code agents in tmux works well, but limits appear around four agents.