On Turn-Level Best-of-N Sampling for Scaling LLM Agents

Who Should Read

An agent developer whose terminal or coding agent loops on failing commands or skips tests, making success rates inconsistent.

Core Mechanics

• Sample N actions per turn, run one
  • A generator model proposes N candidate actions and a selector model picks one to execute.
  • The unchosen candidates are discarded, so they never need sandbox execution.
• Not the same as trajectory-level BoN
  • Trajectory-level BoN runs N full attempts and picks the best afterward, so it cannot change intermediate actions.
  • Turn-BoN intervenes every turn and changes the agent's behavior itself.
• Selector is an off-the-shelf LLM
  • No separate critic was trained; Kimi-K3 and Deepseek-v4.1-Flash were used as is.
  • The selector sees all candidates as a list and returns the best index as JSON.
• Selector prefers doing over inspecting
  • It picked editing files and running tests more often, and read-only actions less often.
  • Empty tool calls and loops on failed commands were executed less.
• Cost is 3–9x tokens
  • Generating N candidates plus a selector call each turn gives 3.6–8.7x total tokens versus baseline.
  • The authors call it closer to a thought experiment than a practical method.
• Selector can pick shortcuts too
  • On the ambiguous mteb-retrieve task, the selector picked candidates that downloaded the reference repository.
  • This was rare among all passing attempts.

Evidence

• Terminal-Bench v2.1 pass@1 gains
  • K2-Horizon-375B went from 73.3% baseline to 77.9% at N=4 and 76.7% at N=8.
  • Qwen3.6-35B went from 47.5% baseline to 52.8% at N=4 and 55.2% at N=8, averaged over two selectors.
• Fewer average turns
  • K2-Horizon went from 69.5 turns at baseline to 41.0 at N=4 and 33.9 at N=8 with the Kimi-K3 selector.
  • Qwen3.6 went from 64.9 turns to 40.5 at N=4 and 34.0 at N=8 with Kimi-K3.
• Less looping and blind retrying
  • Qwen3.6 looping flags dropped 18 points at N=8 from a 42% baseline, per the LLM judge.
  • Qwen3.6 'retried without diagnosis' fell 22 points at both N=4 and N=8 from a 44% baseline.
• Token cost
  • K2-Horizon used 3.6x total tokens at N=4 and 5.6x at N=8 with Kimi-K3, versus a 72K-token baseline.
  • Qwen3.6 used 5.2–8.7x versus baseline. All on 89 Terminal-Bench v2.1 tasks, averaged over four runs.

How to Apply

• Test a selector prompt on candidates
  • Sample 3–4 candidate actions for one agent turn and paste them into the prompt in the code example.
  • Check in the logs whether the rationale favors runnable, verifiable actions.
• Penalize repeated failed commands
  • Add a line to the selector instruction that penalizes candidates repeating a failed action or submitting early.
  • Start with services where agents loop often.
• Apply N only on hard turns
  • Running every turn costs 3–9x tokens, so set a cost cap and enable it only right before submission or right after a failure.
  • The paper leaves partial-turn use as an open question, so compare it yourself.
• Log selector outputs and monitor them
  • Save the selector's rationale in your logs.
  • Check whether shortcuts such as fetching the answer externally get picked.

Try It

# Paste into Claude/ChatGPT: simplified version of the paper's method (acts as a selector choosing among 3 candidates)

You are a private selection reviewer for an autonomous software engineering agent.
Exactly one candidate will be allowed to proceed; the rest are discarded. Pick the candidate most likely to make correct, genuine progress toward completing the task. Prefer concrete, verifiable actions over vague ones. Penalize a candidate that repeats an action already shown not to work or that submits/finishes prematurely. Return exactly one JSON object and nothing else.

=== TASK AND CONVERSATION SO FAR ===
{{task and the [assistant]/[tool] log so far}}

=== CANDIDATE 0 ===
{{tool_call of candidate 0}}
=== CANDIDATE 1 ===
{{tool_call of candidate 1}}
=== CANDIDATE 2 ===
{{tool_call of candidate 2}}

=== REQUIRED JSON RESPONSE SCHEMA ===
{"selected_index": "integer, 0-based index of the best candidate", "rationale": "string, one or two sentences"}

Next step: resample fresh candidates from the same state and repeat, then check whether the selection stays the same.

Terminology

Related Papers

• Claude Code sends your email address to websites

  Claude Code put users' emails in curl User-Agent headers, prompting spam complaints and a built-in blocking mod

• Launch HN: Rivet (YC W23) – Scale and Orchestrate Agents on S3

  Running each agent as an Actor gave 4.99 ms p99 cold starts and 72 KB RAM per instance.

• Show HN: Stroq – a local audit of what your coding agent did and why

  Stroq finds leaked keys in agent sessions and traces risky commands to their source before they run

• Building a Multi-Agent System from Scratch: Architecture Patterns

  Splitting work across agents adds coordination problems, so enforce hard limits in code.

• When Do Summaries Become Instructions? Unsupported Policy in LLM Agents' Self-Written Handoff Summaries

  Making the system prompt's approval sentence specific cut unsupported restrictions in agent summaries from 5.0% to 1.8%.

• AI is forcing us to pay off 40 years of technical debt

  AI turned an old PHP crash bug into a code-execution exploit within two weeks.

• Postman silently auto-installs AI plugins and attempts to share all API data

  Opening Postman auto-installed plugins and a startup hook into Codex and Claude Code

• Octomind – open-source AI harness that saves your tokens, static rust

  A single Rust binary runs any model as a coding agent, with TOML guardrails replacing approval prompts.