WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites
TL;DR Highlight
Fixing the harness, not the model, raised a web agent's challenge score from 31.0 to 57.0
Who Should Read
Agent developers whose screenshot-based browser agent misclicks or repeats actions without an obvious reasoning error
Core Mechanics
- The harness was the problem, not the model
- Each step is a round trip through parsing, execution, feedback, and observation.
- Many live-site failures broke at one of these stages, not in reasoning.
- With the same model (claude-opus-4-6), the score rose from 31.0 to 57.0 over four submissions.
- Every click landed at 3/4 of its target
- The API downscaled screenshots to about 1440x810, but the code treated coordinates as 1920x1080.
- The authors resized images themselves and mapped coordinates by the ratio to the real viewport.
- Clicks that missed by 122-189px landed within 1-5px.
- The model typed its own chat tokens into search boxes
- It emitted chat-template tokens like <|eot_id|> that got typed into inputs.
- They entered history and recurred at every later step.
- Keeping only the first Thought/Action and stripping such tokens fixed it.
- The prompt promised actions the executor never did
Evidence
- Coordinate alignment: 8/24 to 20/24
- Paired comparison on 24 dev tasks: 12 fixed, 0 broken.
- Tasks hitting the step limit fell from 67% to 4%.
- Official score trajectory
- 31.0 (8/29), 41.0 (8/31), 46.0 (9/1), 57.0 (9/2), 2nd place in WebRetriever Challenge 2026.
- The first run finished only 79/100 tasks; about 7 of the first +10 came from completing more tasks and about 3 from accuracy.
- Token-contaminated episodes succeeded less
- 15 of 305 local episodes (4.9%) were contaminated.
- Success was 7.7% for contaminated versus 41.7% for clean, an observational comparison.
- Offline replay of 7,705 steps changed all 551 contaminated steps and none of 7,154 clean ones.
- The goto shortcut inflated scores
- An early dev70 audit found 692 goto calls, and 58% of correct answers came from shortcuts.
- URL navigation was removed from the action set.
How to Apply
- Classify your failures into the four stages
- Today, sort 10 failed trajectories into parsing, execution, feedback, or observation.
- Fix the most common stage before swapping models.
- Measure your screenshot coordinate correction
- Send the model an image with about six known markers and compare its coordinates.
- If they drift, resize screenshots yourself and map by the ratio to the viewport.
- Test prompt-parser-executor consistency
- Automatically verify every action and promise in the prompt is implemented in the parser and executor.
- Strip special tokens and keep only the first action.
- Move budgets and verification triggers into the executor
- Enforce per-action limits in the executor and return the reason on rejection.
- Write rules like 'call read_text before finished' in the prompt.
Try It
# Paste into Claude Code or ChatGPT (simplified version of the paper's method: harness four-stage check)
I'm building a screenshot-based web agent. Look at the failure logs below and classify where each failure broke.
Stages: 1) Parsing (was the model reply turned into the intended action?) 2) Execution (did the action take effect on the page?) 3) Feedback (was the result reported to the model accurately?) 4) Observation (did the model see the information it needed?)
{{failure logs}}
For each failure, give a table with (a) stage (b) evidence it was not a reasoning error (c) one harness fix.
Next to try: paste the action list from your prompt and your executor code, and ask it to find actions that are promised but not implemented.Terminology
Related Papers
Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.
Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents
Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.
Harness Engineering 101: The LLM is the brain, the harness is the body
An agent is a loop managing a JSON array; 16 chapters build one from scratch
Harness2 by Google Research – Recursive Agent Harnessing
Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.
Ousted Savannah police officer gave ICE agent husband access to Flock system
A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired
Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling
Running Claude Code agents in tmux works well, but limits appear around four agents.
Related Resources
Original Abstract (Expand)
We present WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 with a final score of 57.0 out of 100. The challenge evaluates agents end to end on Protocol III of the WebRetriever benchmark (arXiv:2607.06118): starting from an entry URL on a live website, the agent must operate the site's own interface and return a verifiable answer. A capable multimodal large language model (LLM) is necessary for this, but not sufficient. The model's decisions reach the browser through the harness, the code between the model and the page. At every step, four things must go right: the model's reply must be parsed into the intended action, the action must take effect on the page, the result must be reported back accurately, and the model must be shown the information it needs. On real websites, many of the failures we observed occurred at one of these four stages rather than in the model's reasoning. A coordinate-space mismatch placed every click at 3/4 of its intended coordinates; actions on native dropdowns, inside iframes, and in text boxes failed silently; and self-generated chat-template tokens contaminated 4.9% of task episodes. WebFovea hardens each stage and surrounds the loop with guardrails that keep the agent within the rules and its budget. The four-stage view does not depend on the model, although some individual fixes do. Because we used the same model in all four submissions, the rise of our official hidden-set score from 31.0 to 57.0 reflects changes to the harness, up to run-to-run variance on live sites. We describe the design, the evidence for each component (including negative results), a failure analysis, the limitations, and a roadmap that includes routing different steps to different models. Code is available at https://github.com/jianganghan/WebFovea.