Improving Environment-Agnostic Agents: The Role of Scaffolding
TL;DR Highlight
One prompt line telling the agent to stop and report raised game scores, but cost more tokens.
Who Should Read
Agent developers running Codex-style coding agents in loops who see them stall or give up early.
Core Mechanics
- Only one prompt line changed
- Baseline said "Continue working until you have completed your task."
- Stop-and-report said to continue only while confident and to stop and report on problems or ambiguity.
- Code, tools, and controller stayed identical.
- A controller resumes after each report
- When Codex stops, a scripted controller resumes the same conversation.
- It adds only a goal reminder and a nudge to reread the environment description, with no human feedback or hints.
- Two games improved, Terminal-Bench was mixed
- Mean scores rose in every setting on ARC-AGI-3 and AGI Maze.
- Terminal-Bench changed by -1.50 to +2.25 points with no consistent direction.
- Higher scores came with higher token cost
- Cost rose at the same reasoning effort.
- The experiments cannot separate better use of information from simply using more inference.
Evidence
- ARC-AGI-3 RHAE rose 1.37-7.29 points with Stop-and-report vs Baseline
- GPT-5.5 high went from 27.57 to 34.86.
- GPT-5.6-sol max went from 89.59 to 90.96.
- AGI Maze RHAE70 was higher in all 8 comparisons
- Gains were 0.37-4.95 points, with token cost up about 7-39%.
- GPT-5.6-sol max with tutorials went from 68.03 to 72.98.
- Controlled_RESET recovered about 34-60% of the gap
- The gap is between Stop-and-report and twma_v1.61.
- GPT-5.6-sol xhigh went from 77.33 to 84.59, vs 92.24 for twma_v1.61.
- Cost rose in the same order as score
- On ARC-AGI-3, Stop-and-report cost about 2-52% more than Baseline.
- GPT-5.5 high went from 9.36M to 14.19M, and twma_v1.61 used 64.09M.
How to Apply
- Swap your stop instruction for Stop-and-report
- Replace your "keep going until done" line with the paper's report instruction.
- Run about 10 identical tasks before and after and compare success and tokens.
- Build an auto-resume loop for stopped agents
- When the agent stops, resume the same conversation with only a goal reminder.
- Add no human hints, and exit after two consecutive returns with no action.
- Let the controller execute rules already in your prompt
- For clear-cut rules like retry or reset, forbid the agent from calling them.
- Have a script run the action when the condition is detected.
- Track score and cost together
- Log success rate and token usage for every prompt change.
- Judge whether the gain is worth the added cost.
Try It
# Paste at the end of your Claude Code or Codex system prompt (simplified version of the paper's method)
# When to stop
Continue working on the task as long as you can proceed confidently. If you encounter a problem, ambiguity, unexpected issue, or anything that makes you unsure how to proceed, stop and report it back to me before taking further action.
# Message for the controller to send when resuming a stopped agent
Keep working toward the goal described in {{goal description file}}. Re-read the file whenever you need a reminder.
Next step: compare against "Continue working until you have completed your task." on the same tasks, and record token usage too.Terminology
Related Papers
Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.
Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents
Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.
Harness Engineering 101: The LLM is the brain, the harness is the body
An agent is a loop managing a JSON array; 16 chapters build one from scratch
Harness2 by Google Research – Recursive Agent Harnessing
Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.
Ousted Savannah police officer gave ICE agent husband access to Flock system
A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired
Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling
Running Claude Code agents in tmux works well, but limits appear around four agents.
Related Resources
Original Abstract (Expand)
Agents built on stronger language models may increasingly construct useful procedures for themselves, reducing the advantage of developer-supplied scaffolding. We investigate why this advantage persists and whether an environment-agnostic agent can be improved without environment-specific engineering. We evaluate a Codex-based agent in two nearly identical prompt variants on ARC-AGI-3, AGI Maze, and Terminal-Bench. Each variant is deployed with unchanged code, prompts, and general-purpose tools across environments, receiving environment-specific information as input. Changing only an instruction to encourage reporting difficulties and returning control for automatic continuation improves mean scores on both ARC-AGI-3 and AGI Maze, while Terminal-Bench shows small, mixed changes. Yet the general agent remains below an ARC-AGI-3-specific reference, including with GPT-5.6-sol. The reference prescribes organization and control rather than supplying new game rules or solutions. To investigate this gap, we transfer reset decisions to a scripted controller that follows a convention already given to the agent. This small change recovers part of the specialized agent's advantage. A complementary AGI Maze experiment shows that additional tutorial experience can reduce performance. Together, these findings highlight the distinction between having information and tools available and using them effectively. Supplied procedures still matter for the models tested here. The performance gains also come with higher token-based cost. The experiments do not separate better use of information and procedures from additional or differently allocated inference. We provide the EnvTraverse framework and an evaluated baseline agent for testing whether further environment-independent changes improve performance in some environments without degrading it in others.