An empirical study of harness design for coding agents

TL;DR Highlight

Varying harness components one by one shows the best setup depends on model and context budget.

Who Should Read

A backend developer deciding how to set up context management, planning, and tools for a coding agent

Core Mechanics

Context management matters more as budgets tighten
- The tighter the context-window budget, the more context management is worth.
- Most of the benefit comes from preventing context-overflow failures.
Rule-based elision first, then LLM summary
- Staging rule-based elision before LLM summarization was the most efficient of the five strategies.
- Making elided content recoverable added machinery that models rarely used and gave no accuracy gain.
Planning: accuracy for weak models, savings for strong ones
- For weaker models, planning acts as an accuracy scaffold.
- For stronger models, accuracy barely changes and planning mainly cuts cost.
Bash-capable models can go bash-only
- Models with weak bash skills do better with predefined tools.
- Bash-capable models work well with a bash-only interface at substantially lower cost.
- The gap was largest on command-line-centric tasks.

Community Reactions

Opinion that Nemotron and Mistral are far from the frontier
- Commenters said the models tested (Nemotron family and Mistral) don't reflect cheap Chinese models or large Claude/GPT models.
- Missing Qwen and DeepSeek was called a miss.
Opinion that frontier models don't need planning tools
- A commenter shared a Claude Code issue saying todo/task tools are off by default on newer models and need an env variable.
- They read it as Anthropic agreeing that frontier models don't need in-session planning tools.
Experience checkpointing at 25-30% of the window
- One commenter writes state to PROGRESS.md and a requirements JSON at 25-30% of the window and restarts fresh.
- They cited the paper's gap of 35.7 points at 32k versus 2.7 points at 128k.
Opinion that simple agents often beat complex harnesses
- The mini-swe-agent author said few benchmarks show complex harnesses consistently beating very simple agents.
- Another commenter said using a harness and model from the same lab gave a far better experience than general-purpose harnesses.

How to Apply

Cut long sessions at 25-30% and resume from a state file
- In today's agent session, have it save progress to PROGRESS.md once context hits a set fraction.
- Open a new session and resume from only that file.
Add rule-based trimming before LLM summarization
- Trim old tool outputs by rule first.
- Add LLM summarization after that only if still needed.
- Skip recoverable-elision tools for now.
A/B bash-only versus predefined tools per model
- Run the same task set with bash-only and with predefined tools and compare success rate and cost.
- If success is equal, cut the tools.
Try dropping planning on strong models
- Run the same tasks with planning on and off and compare token cost and success rate.
- If success is similar, turn planning off to save cost.

Try It

copy & run

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: harness component audit)

I run a coding agent harness. Review the setup below and point out components that are unnecessary or missing for this model and budget.

- Model: {{model name}}
- Context window budget: {{e.g. 32k / 128k}}
- Current context management: {{e.g. none / summary / elision}}
- Planning step: {{yes / no}}
- Tool list: {{tool list}}
- Main task types: {{e.g. shell-centric / code editing}}

Answer in this order.
1. Is there a risk of context overflow failures, and how to restructure so rule-based elision comes first and LLM summarization after
2. Whether planning serves accuracy or cost savings for this model, and whether it can be removed
3. Whether I can cut down to bash-only, or why predefined tools are needed
4. An A/B experiment plan to verify the three points above (metrics: success rate, token cost)

Next step: ask it to run 10 tasks per configuration and tabulate success rate and cost.

Terminology

harnessThe wrapper around an LLM that handles tool calls, the execution loop, and context management. In a car analogy, the wheels and gearbox rather than the engine.

context managementShrinking or tidying conversation and tool output so the context window doesn't overflow. Like clearing papers off your desk before it fills up.

elisionCutting or omitting old content by rule. Unlike summarization, it needs no LLM call.

action spaceThe set of tools and commands an agent can use, such as bash only versus dedicated file-edit tools.

SWE-Bench VerifiedA benchmark measuring coding ability by having agents fix real open-source issues.

Terminal-BenchA benchmark measuring how well agents complete tasks in a terminal environment.