Scaling Inference-Time Compute with Teams of Coding Agents | AI Paper Digest

TL;DR Highlight

Delegating only planning to workers matched full-delegation reward at half the cost.

Who Should Read

An agent developer building multi-agent coding systems whose costs jump 3–7x with teams.

Core Mechanics

Teams score higher but cost 3–7x
- A team of all strong agents beats a solo agent on reward.
- Inference costs 3–7 times as much, and most configurations take longer.
Spend workers on planning first
- A lead that delegates only planning and implements and verifies itself matches the reward of delegating all stages at half the cost.
- Planning workers probe the black-box executable in parallel and bring back information.
Cheap models work only as planners
- A strong lead with cheap planners recovers 47.5% of an all-strong team's reward gain for 13.9% of its added cost.
- Cheap implementers lower reward.
The lead sets the ceiling
- With workers fixed at Luna, reward rises with each lead upgrade.
- Cheap workers cannot make up for a weak lead.
Only planners keep scaling with team size

Evidence

On 200 ProgramBench tasks, Sol team efficiency η was 3.8 for planning and at most 2.4 for other stages
- η is reward points gained per $100 of added cost over the solo agent.
- A dollar on planning buys about 1.6x the reward of other stages.
A Sol lead with 16 Luna planning workers came within 1.4 points of an all-Sol team at 70% lower cost
- This is under planning guidance, reported as roughly 4x more efficient.
- Fixed-size teams are single runs.
Luna planners had η=51.0 vs 3.8 for Sol planners (blocking)
- Luna implementers had η=−12.0, so each extra dollar lowered reward.
- Versus PIV, planning raised reward 5.6% and cut cost 22.2%.
Upgrading the lead raised reward 20.9% (Luna→Terra) and 15.4% (Terra→Sol)
- Workers were fixed at Luna.
- A Terra lead with Luna workers still trailed a solo Sol by 10.3%.

How to Apply

Add a planning-only delegation rule to your lead prompt
- Add "use sub-agents only for exploration and planning; implement and verify yourself" to your current lead system prompt.
- Tell workers to investigate and report without editing code.
Swap only the workers to a cheaper model and compare
- Keep a strong lead and switch only planning workers to a cheap model, then measure reward and cost on the same task set.
- Do not switch implementation workers to cheap models.
Scale planners first when adding workers
- Cap implementers and verifiers at around four.
- Grow planners through 2, 4, and 8 and plot the reward-cost curve.
Turn off inter-agent messaging
- Do not give agents a send_message-style tool by default since it showed no reward gain.
- Return worker results to the lead only when workers finish.

Try It

copy & run

# Paste into Claude Code or ChatGPT (simplified version of the paper's method: a planning-only delegation lead prompt)

You are the lead of a team of engineers. Solve the task in this order.
1. Use sub-agents only during planning and exploration. Split into non-overlapping investigation areas (requirements and interfaces, major features, edge cases, implementation strategy and risks) and delegate them concurrently. Give each worker its question, known context, and report format, and tell it not to edit code.
2. After collecting all reports, merge them into one prioritized implementation plan.
3. Once the plan is done, stop delegating and implement and verify yourself.

Task: {{task description}}

Next, try: run the same task with "use sub-agents in every stage" and compare cost and results.

Terminology

Pareto frontierThe line connecting settings that give the best performance at each cost. A setting off this line has a cheaper or better alternative.

Lead-worker topologyA lead agent splits the task, hands pieces to worker agents, and merges their results.

PIVPlan, Implement, Verify. PIV guidance tells the lead to use workers in all three stages.

Blocking vs. async delegationBlocking means the lead waits until the worker finishes. Async means the lead keeps working and receives results later.

ProgramBenchA benchmark of 200 tasks where an agent rebuilds a program from scratch using only its executable and docs.

Related Papers

Original Abstract (Expand)

Multi-agent systems promise a new axis for scaling inference: rather than having one agent think longer or sample more attempts, a task is split across many agents that work in parallel and coordinate. However, in practice, more agents can easily produce duplicate work, coordination failures, and higher costs. We therefore formulate team design as a compute allocation problem: starting from a solo agent, where should additional inference compute go to reach the reward-cost Pareto frontier? We build a minimal multi-agent harness from the ground up and trace this frontier by varying, one at a time, the workflow stage at which a lead delegates to its workers (planning, implementation, or verification), the models that form the team, and the number of workers. On ProgramBench, teams outperform solo agents but cost three to seven times as much. With the right compute allocation, however, a heterogeneous 17-agent team comes within 1.4 points of a strong homogeneous 5-agent team at 70% lower cost. This efficiency stems from three findings. First, planning is the most valuable use of workers: in a homogeneous team whose lead decides how many workers to spawn, a lead that delegates only planning to its workers, and then implements and verifies the code itself, matches the reward of a lead that delegates to workers throughout planning, implementation, and verification, at half the cost. Second, cheap models pay off as planners but not as implementers: a strong lead with cheap planners recovers 47.5% of the reward gain of an all-strong team for only 13.9% of its additional cost, whereas cheap implementers lower reward. Third, planners keep improving reward as the team grows, whereas implementers and verifiers plateau beyond four workers. Together, our results offer a guide to spending an agent budget where it counts, given the task, cost, and latency constraints a user has.