CUAWright: A Minimal Unified Interface for Digital Agents
TL;DR Highlight
Giving agents only bash instead of GUI raised OSWorld 2.0 scores and cut cost 37.5%.
Who Should Read
A developer building browser or desktop agents on screenshot-click loops who is tired of high cost and frequent failures.
Core Mechanics
- One bash action replaces everything
- Browser tasks run as code against the debugging protocol (CDP).
- Mouse and keyboard via xdotool/pyautogui is used only when a surface blocks programmatic control.
- Screenshots only on demand
- The default observation is terminal output.
- The agent captures a screenshot and loads it with two separate CLI calls when needed.
- The file system is memory and toolbox
- Extracted values, progress and failure logs are saved as files and survive context compaction.
- The agent writes scripts and reuses them later.
- Agents built their own tools
- On OSWorld 2, GPT-5.6 Sol saved code or executable files in 84.8% (89) of tasks.
- Of 776 executables created, 24.7% (192) were reused.
- Multi-agent didn't help much
- Adding planner and verifier agents performed about the same as a single agent.
Evidence
- OSWorld 2.0 cost and score
- GPT-5.5 xhigh: 47.5% official vs 63.2% CUAWright (+15.7pt), cost per task $16.1 to $10.1.
- GPT-5.6 Sol max: 62.7% official vs 67.9% (+5.2pt), cost per task $13.9 to $4.4.
- Web benchmark success rates
- Online-Mind2Web: GPT-5.4 GUI 83.4% vs CUAWright 88.1% (+4.7).
- Odysseys: GPT-5.4 GUI 33.5% vs 77.5% (+44.0), averaging 76.1 steps.
- File system ablation
- On OSWorld 2.0, GPT-5.6 Sol partial score dropped from 67.9% to 55.5% (-12.4pt).
- The text also states -12.1 pts.
- WeaveBench and CAD
- WeaveBench PassRate with GPT-5.5: Codex CLI 35.1% vs 53.5%.
- BenchCAD with-tool mean IoU: official GPT-5.5 55.8% vs 79.0%.
- CADGenBench overall: Codex+MCP tools 45.7% vs 49.4%.
How to Apply
- Add a workspace-file rule to your prompt
- Add to your browser or desktop agent's system prompt: keep notes in a workspace file.
- Tell it to save repeated steps as scripts and compare success rates.
- Drop screenshots as the default observation
- Switch observations to terminal output or accessibility-tree text.
- Split image handling into capture and load commands called only when needed.
- Replace coordinate clicks with CDP code
- Pick form or list tasks where coordinate clicking often fails.
- Let the agent write a CLI script that finds elements by accessibility name and clicks them.
- Set up sandbox and rollback first
- Run the agent inside a container or VM before it can edit files or settings.
- Take a snapshot before changes so failures can be reverted.
Try It
# Prompt to paste into Claude Code or ChatGPT (simplified version of the paper's method)
You are an agent that works only through the terminal. Task: {{task description}}
Rules:
1. Every action is a bash command. Use screenshots only when truly necessary.
2. Keep updating ./workspace/notes.md with extracted values, progress, and failure records.
3. If you do the same operation more than twice, make it a parameterized script under ./workspace/tools/ and reuse it.
4. Before finishing, re-read the result directly to verify it.
First inspect the environment, then run commands; do not plan without acting.
Next try: compare step count and token cost against a GUI-style prompt on the same task.Terminology
Related Papers
Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.
Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents
Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.
Harness Engineering 101: The LLM is the brain, the harness is the body
An agent is a loop managing a JSON array; 16 chapters build one from scratch
Harness2 by Google Research – Recursive Agent Harnessing
Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.
Ousted Savannah police officer gave ICE agent husband access to Flock system
A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired
Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling
Running Claude Code agents in tmux works well, but limits appear around four agents.
Original Abstract (Expand)
The prevailing approach to computer-use agents couples a model with a domain-specific harness: a browser or desktop environment equipped with human engineered tools that are fixed before task execution. As models' coding capabilities improve, the GUI native and static harness prevents them from direct programmatic operation on system state, as well as flexible construction of tools. To this end, we introduce CUAWright, a minimal terminal harness of roughly 3K lines of code that uses bash commands as its sole action interface, and a file system as its evolvable space for dynamically creating tools and managing the context. We conduct comprehensive experiments across a wide range of digital tasks, and demonstrate that by giving the agent a minimal, programmable interface, it achieves substantially stronger results compared to their GUI or hybrid CLI interface across a wide range of tasks. On OSWorld 2.0, CUAWright delivers a 33.2% relative improvement in partial reward while reducing estimated cost by 37.5% compared with the published GPT-5.5 baseline. On Online-Mind2Web and the long horizon Odysseys benchmark, CUAWright substantially outperforms GUI native harness by 4.7% and 44.0% in success rate, respectively. Furthermore, we found the gains extend to CAD applications that require accurate visual understanding and CLI interaction: on CADGenBench and BenchCAD, our unified harness yields 8.1%-41.6% relative improvements over other CLI based harnesses with GPT-5.5. Together, these results suggest digital environments are far more programmable than their GUI interfaces imply, and a minimal terminal-focused harness is the key for better performance and efficiency.