Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement | AI Paper Digest

TL;DR Highlight

Running coding agents in planner-developer-QA loops for three iterations raised GameCraft-Bench scores by up to 22 points.

Who Should Read

A developer whose coding agent (Codex, Claude Code, etc.) loses direction on large tasks and wants a reliable long-running workflow.

Core Mechanics

Split the coding agent into three roles
- HoH reuses an existing harness (the shell that manages an agent's tools and execution) and invokes a Planner, Developer, and QA Tester separately.
- Each loop builds and verifies one small capability.
Separate the builder from the checker
- QA gets a frozen, read-only candidate and judges it on builds, runs, screenshots, and logs.
- The developer's claim of being done does not count as acceptance.
Pass both code and test evidence to the next loop
- Code alone forces the next loop to re-infer what is broken and why.
- Verified behavior becomes a preserve constraint and failures become repair targets.
It beats simply running more passes
- At the same number of passes, it scored higher than just prompting the agent to continue.
- At some budgets it also used fewer tokens for a higher score.
Removing any mechanism hurts

Evidence

All three configs improved on GameCraft-Bench after 3 loops
- Codex+GPT-5.5 (high) 49.58→71.52, OpenCode+DeepSeek-V4-Pro 26.90→48.98, Pi+MiniMax-M3 42.16→58.78.
- Each is versus a single Vanilla run of the same harness and model.
FrontierSWE and ProgramBench also improved
- FrontierSWE mean reward: Codex 0.31→0.54, OpenCode 0.23→0.31, Pi 0.26→0.55.
- ProgramBench avg test pass rate: Codex 60.41→66.50, OpenCode 45.27→57.56, Pi 35.83→52.68.
Better than naive repetition at equal passes
- At 3 passes on GameCraft-Bench, HoH scored 71.52 (8.41M tokens) versus 58.24 (6.33M tokens) for Vanilla Continuation.
- HoH at 2 passes scored 64.84 with 5.67M tokens, beating 3-pass Vanilla Continuation.
Gains continued over 10 loops on FrontierSWE
- With Codex+GPT-5.5, Dominance rose from 39.33% at HoH@3 to 72.67% at HoH@10.
- Vanilla was 27.33%.
Every ablation lowered the score

How to Apply

Write a dedicated QA prompt today
- Tell it not to modify code and to judge only from runs, screenshots, and logs, labeling items verified or gap.
- Run it in a separate session right after the coding agent says it's done.
Save an evidence file each loop and feed it to the next plan
- Store verified items, gaps, things to preserve, and things to fix as JSON or Markdown.
- Give the next Planner only the requirements and this file.
Cap each loop at three priorities
- Have the Planner rank regressions and build blockers before feature work.
- Attach an observable validation condition to each priority.
Enforce output schemas and retry on violations
- Fix only each role's output format and leave the approach to the agent.
- Automatically re-invoke when the format is wrong.

Try It

copy & run

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: QA role only)

You are the QA Tester. Do not modify code. Judge only from observable evidence such as run results, logs, and screenshots, against the requirements and this iteration's development document below.

[Requirements]
{{요구사항}}

[Development document for this iteration]
{{개발 문서}}

[Developer's output / run logs]
{{결과물 또는 로그}}

Split the requirements into checkable claims. Mark a claim verified only when the evidence is sufficient. Record failures, regressions, and insufficient evidence as gaps. Answer only in this JSON.
{"verified":[{"claim":"","evidence":""}],"gaps":[{"claim":"","evidence":"","player_impact":"","recommended_update":""}],"preserve":[""],"next_targets":[""]}

Next step: give this JSON to a Planner in a new session and have it write the next plan with at most three priorities, each with an observable validation condition.

Terminology

harnessThe wrapper system that gives an LLM tools and manages execution and results. Codex CLI and OpenCode are examples.

warm-startStarting the next iteration from the previous artifact instead of an empty project. It is like resuming work in files you left open.

shift-left testingRunning tests right after each change rather than at the end of development. It is like running unit tests as soon as you write code.

progressive disclosureShowing only an index first and fetching details when needed. HoH uses a file-system index instead of a memory module.

DominanceOn FrontierSWE, the share of per-task comparisons a configuration wins against randomly chosen other configurations.

Related Papers

Related Resources

https://github.com/Flesymeb/HarnessOfHarness

Original Abstract (Expand)

This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/