LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks | AI Paper Digest

TL;DR Highlight

Keeping task state outside execution and auditing it lifted WeaveBench from 51.8% to 80.7%.

Who Should Read

Developers running long Claude Code or Codex agent tasks who see repeated mistakes or premature 'done' claims

Core Mechanics

Execution, state, and completion judgment share one context
- Existing harnesses run the task, track state, and judge completion in the same growing session.
- So a wrong completion judgment becomes a premise for later decisions.
Manager-Executor-Auditor loop
- The manager cannot touch the environment and writes a subtask contract (goal, acceptance criteria, constraints) from state and audit reports.
- The executor does only that subtask in a fresh context and its history is discarded.
- A read-only auditor inspects the environment directly and writes a report.
Executor claims never enter the state
- A record becomes completed only with clean audit evidence.
- If the auditor mutates anything, it is logged as an integrity violation and the report cannot support completion.
Audit caught a plausible-looking completion
- The baseline edited XML directly because headings looked right and scored 0.00.
- With the harness, the auditor parsed the XML and confirmed 15 heading styles, scoring 0.89.

Evidence

WeaveBench: Qwen 3.7-Plus pass rate 51.8% to 80.7%
- Versus the same-model Claude Code baseline on 114 tasks.
- Higher than the official 41.2% of Claude Opus 4.7 with Claude Code, though that run used different privileges and is reference only.
Terminal-Bench 2.1: 69.7% to 77.2%
- Versus the Qwen 3.7-Plus + Claude Code baseline.
- With Codex backend and GPT-5.6 Luna it reaches 83.1%, and used 24% fewer tokens than the baseline.
OSWorld 2.0 binary 2.8% to 8.3%, partial 21.5% to 35.2%
- Versus Qwen 3.7-Plus single action on 108 tasks.
- Average output tokens rose from 28.9K to 104K.
Claude Opus 4.7 on a 34-task OSWorld 2.0 subset: 20.6% to 35.3%
- Versus Opus 4.7 single action, binary metric.
- On 17 WeaveBench Games tasks, Qwen with the harness averaged 0.733, above Opus with plain Claude Code at 0.680.

How to Apply

Run a verification prompt instead of trusting 'done'
- After an agent task today, open a fresh session with only the goal and output location and have it inspect read-only.
- Do not pass the execution log, and ask for pass/fail per criterion with evidence.
Write a contract for each subtask
- Split long work into contracts with goal, acceptance criteria, constraints, and needed prior evidence, and run one at a time.
- Write a check command for each criterion, like 'ls shows the file'.
Keep a state file and mark completed only when verified
- Track requirement, artifact, and fact as pending/completed/blocked/untrusted, and keep executor claims pending.
- Start each round in a fresh context and pass only this file and audit reports.
Set round budgets to control audit cost
- Cap time per round. The paper used 1800s for the executor, 300s for manager and auditor, and at most 25 rounds.
- Try it first on long dependent tasks, since gains are small when a single-step skill is the bottleneck.

Try It

copy & run

# Paste into a fresh Claude or ChatGPT chat (simplified version of the paper's method: a read-only auditor role)

너는 읽기 전용 감사자다. 파일을 수정하거나 상태를 바꾸는 명령은 실행하지 마라.
아래 원래 목표와 합격 기준만 보고, 결과물이 실제로 충족하는지 판단해라. 작업자의 완료 주장은 증거가 아니다.

[원래 목표]
{{원래 목표}}

[합격 기준]
{{합격 기준 목록}}

[결과물 위치/내용]
{{결과물 또는 경로}}

각 합격 기준마다 다음 형식으로 답해라.
- 상태: complete / incomplete / blocked
- 직접 확인한 증거
- 남은 빈틈
마지막에 무결성 상태(clean / suspect / violation)와 다음에 할 서브태스크 하나를 제안해라.

Next: copy the report into your state file, then give a fresh-session executor only the next subtask and repeat.

Terminology

HarnessThe system layer around a model that organizes prompting, tools, context management, and multi-step execution. Claude Code and Codex CLI are examples.

Long-horizon taskA long task needing many interdependent steps. The hard part is staying coherent to the end, not any single step.

Context rotPerformance drops as history grows because relevant info gets hard to find. Like a drawer so full of junk you can't find anything.

Task stateA record of requirements, finished work, produced artifacts, and facts found in the environment. Think of it as a project checklist.

AgentAdapterA thin layer that plugs existing agents like Claude Code or Codex CLI into each role without changing their internal loops.

Fresh contextStarting a run with no earlier chat history, only the information needed for the current round.

Related Papers

Related Resources

Original Abstract (Expand)

Large language model (LLM) agents increasingly undertake long-horizon tasks that require sustained reasoning, tool use, and revision across many interdependent steps. However, existing agent harnesses maintain task execution, task state, and completion assessment within a growing context, making the state difficult to track and allowing incorrect self-assessments to propagate into later decisions. We reformulate long-horizon execution as a task-state management problem and propose LongHorizon-Harness, which maintains the task state explicitly outside execution and updates it only with facts independently verified from the environment. Its Manage-Execute-Audit(MEA) loop uses a manager to maintain the task state and determine the next subtask, a fresh-context executor to perform it, and a read-only auditor to verify the resulting environment state before the next round. A lightweight AgentAdapter supports interchangeable model and harness backends without modifying their native agent loops. LongHorizon-Harness improves Qwen~3.7-Plus from 51.8% to 80.7% on WeaveBench, from 69.7% to 77.2% on Terminal-Bench~2.1, and from 2.8% to 8.3% on OSWorld~2.0. It also raises Claude Opus~4.7 from 20.0% to 34.3% on an OSWorld2.0 subset, demonstrating consistent gains across models, harnesses, and interaction domains.