Do AI Agents Know When a Task Is Simple? Toward Complexity-Aware Reasoning and Execution | AI Paper Digest

TL;DR Highlight

Estimating task scope first and expanding only on failure cut agent cost by 85%.

Who Should Read

Agent developers whose coding agents read every file for trivial edits, wasting tokens and time

Core Mechanics

Agents over-read even on easy tasks
- The paper says agents default to a maximum-context-first habit when uncertain.
- It opens with a two-line icon swap that gets treated like a small code audit.
E3: Estimate, Execute, Expand
- The agent first makes a cheap estimate of difficulty, scope, risk, and confidence.
- It runs a minimal path (locate, edit, verify) and widens scope one level only when verification fails.
Optimistic estimates are fine
- Expansion acts as the safety net when the estimate is wrong.
- Removing Expand drops success to 85.1%, while removing Estimate keeps success but raises cost by 20%.
Simpler tasks waste proportionally more
- Read-everything policies show the highest redundancy ratio (ACRR) on the easiest tasks.
- The authors admit this is partly a mechanical effect of the ratio.
The effect is much smaller on real gpt-4o

Evidence

85% cost cut in the simulator
- On 121 MSE-Bench tasks, versus Max-Context-First, cost fell 84.9%, tokens 90.9%, and inspected files 92.2%, both at 100% success.
- Mean cost dropped from 122.85 to 18.55.
Versus a strong adaptive baseline
- Versus Adaptive Retrieval, cost fell 16.0% and inspected files 66.8% at the same 100% success.
- E3 used about 14% more tool calls.
Fixed ReAct is cheap but unreliable
- Its success rate was 66.9%, and it failed every Level 3 task.
- It never follows imports to the indirect edit site.
Success holds under reworded instructions
- With paraphrased instructions, estimator accuracy fell from 85.1% to 66.9%.
- E3 stayed at 100% success, and cost rose only 8.7%, from 18.55 to 20.17.
gpt-4o results
- Over 5 toml tasks with 3 runs each, E3 used 18% fewer tokens and 18% lower latency than the thorough agent, and 4% fewer tokens and 5% lower latency than ReAct.

How to Apply

Add an estimate step to your agent system prompt
- Today, add "first estimate scope, difficulty, and risk, then start with the minimal path" to your coding agent prompt.
- Run a few identical tasks and compare token counts and files opened before and after.
Expand scope only when verification fails
- Use test or lint failures as the trigger to raise the scope level one step at a time.
- Set a cap K on expansions and reuse earlier search results.
Measure waste per task
- Log actual tokens and files opened against a minimum needed per task.
- Check whether the waste ratio is larger on easy tasks.
Keep an expansion path for hidden dependencies
- In repos with aliases or re-exports that grep misses, include import tracing in the higher scope levels.
- Let expansion recover what the estimate missed.

Try It

copy & run

# Paste into Claude Code or ChatGPT as a system prompt or first instruction (simplified version of the paper's E3 method)

너는 코딩 에이전트다. 아래 순서로 일하라.
1. Estimate: 작업을 읽고 난이도(1~3), 수정할 파일 범위, 위험, 확신도를 한 줄로 적어라. 파일을 전부 읽지 마라.
2. Execute: 가장 작은 경로로 시작하라. 키워드를 검색해 해당 위치만 열고 수정한 뒤 테스트로 검증하라.
3. Expand: 검증이 실패하거나 확신도가 낮을 때만 범위를 한 단계 넓혀라. 직접 수정 위치를 모두 보고, 그래도 안 되면 import와 re-export를 따라가라. 이전에 찾은 검색 결과는 재사용하라. 확장은 최대 {{최대확장횟수 예: 3}}번까지 한다.

작업: {{수정 지시}}
저장소 경로: {{경로}}

Next step: run the same task with a "read everything first" prompt and compare token counts and files opened.

Terminology

ACRRAgent Cognitive Redundancy Ratio. It measures how much actual cost exceeds the oracle minimum. 0 means no waste, and 4 means five times the needed cost.

Maximum-context-firstGathering as much context as possible before acting when uncertain. It is like pulling every item out of the house before moving a single box.

Initial operating pointA starting estimate that is not the answer but keeps the search short and stable. It is like a navigation app sketching a rough route first.

OracleA reference that knows the minimum path for each task in advance. Think of a grader holding the answer key.

MSE-BenchThe paper's deterministic offline benchmark of 121 edit tasks. Each task has a defined minimum-sufficient path.

Related Papers

Related Resources

Original Abstract (Expand)

Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires. They often follow a maximum-context-first strategy--re-reading files and dependencies they have already seen--turning a one-line edit into a small code-base audit. We argue the missing capability is task-aware execution-scope estimation: judging a task's difficulty, the information it truly needs, and the shortest reliable path before committing budget. We formalize minimum-sufficient execution and the Agent Cognitive Redundancy Ratio (ACRR), and propose E3 (Estimate, Execute, Expand): the agent estimates an initial operating point, executes a minimum viable path, and expands scope only when verification fails. On MSE-Bench--a deterministic benchmark of 121 edits in a capability-controlled simulator--E3 matches the strongest baseline's 100% success while cutting cost by 85%, tokens by 91%, and inspected files by 92%, and further beats a strong adaptive retrieval baseline by 16%; the gains survive held-out instruction wording and essentially every cost weighting. A companion real-model harness (LLM-Case) corroborates the effect on a live gpt-4o agent editing a real open-source library, with every candidate patch graded by actually running the project's real pytest suite against a measured oracle: the over-reading is milder but real, and E3 is the leanest and fastest policy at comparable task success--its one shortfall a provider rate-limit, not a wrong edit. We frame this as a controlled probe of execution redundancy, not a measurement of any deployed agent, and position task-aware execution as a step toward engineering-grounded AI (EGAI)--agents whose effort is anchored in the engineering reality of the task. We release the framework and benchmark.