Know When to Zoom: How Small Is Too Small for Vision-Language Models?
TL;DR Highlight
Splitting large photos into tiles raised Claude and Gemini accuracy by 11.1–14.5 points
Who Should Read
A backend developer analyzing drone, satellite, or CCTV photos through Claude, Gemini, or GPT APIs who keeps losing small objects
Core Mechanics
- Services shrink images before the model sees them
- Claude shrinks the long side to at most 2576 px, and Gemini converts images to 280–2,240 tokens.
- OpenAI's default rejects images needing over 30,000 patches with an error.
- Bigger photos need bigger text to be read
- On the smallest canvas every model read text of 6 px or less.
- On a 7680×4320 photo at default settings, text needs a predicted 10–13 px to be read half the time.
- Tiling wins the detail back
- Up to nine tiles raised Claude and Gemini accuracy by 11.1–14.5 points.
- GPT-6.1 Sol went from 50.7% to 83.7% by avoiding rejections.
- Two-step zoom uses far fewer tokens
- The model first names the region it needs, then sees that crop next to an overview.
- On Gemini it matched or beat tiles with about 40% of their image tokens.
- Gemini stays confident when wrong
- When detail was lost, both Claude and Gemini said they couldn't tell about half the time.
- Of the rest, confident wrong answers were 43–48% for Gemini versus 5–9% for Claude.
- On Claude, the reply itself is the zoom signal
- A detail-size check before the call missed most of the tile gain.
- Tiling only after an unsure reply came within 1.4 points of always tiling.
Evidence
- Tiling gains on 601 questions
- Versus sending as is: Haiku +14.5, Sonnet +11.1, Flash-Lite +12.0, Flash +12.5 points.
- On remote sensing, Sonnet went from 60.0% to 80.8% with 3×3 tiles.
- OpenAI default rejections
- 215 of 601 photos were rejected, including 144 of 150 HR-Bench questions.
- On the 386 accepted photos, Sol scored 79.0% as is.
- Two-step zoom accuracy versus 3×3 tiles
- Flash-Lite 72.2% vs 70.5%, Flash 83.0% vs 80.5%.
- Luna scored 4.5 points above tiles.
- Reply-based trigger results
- Haiku reached 72.4% and Sonnet 82.9%, within 1.4 points of always tiling.
- It used 62–70% of always-tiling's image tokens.
How to Apply
- Check your pipeline's resize rules today
- List your provider's image limits (Claude 2576 px, Gemini token cap, OpenAI 30,000 patches).
- Compare them with your real photo sizes and count how many get shrunk or rejected.
- On Claude, tile only after an unsure reply
- Call with the original first, and re-call with 2×2 or 3×3 tiles if the answer says it can't tell or confidence is under 70.
- Tile every photo where accuracy matters most.
- On Gemini, make two-step zoom the default
- Ask in call one for left, top, right, bottom as 0–1 fractions and validate the box.
- If invalid, send the whole photo; otherwise crop with 20% margin and send it with an overview.
- On OpenAI's default, tile large images
- Tile any image above 30,000 patches because it gets rejected.
- Keep zoom crops under that limit too.
Try It
# Paste into Claude or ChatGPT with the large image attached (simplified version of the paper's two-step zoom)
I attached a large photo and want to answer a question about it.
Question: {{질문}}
Step 1: Before answering, tell me which region of the photo I should zoom into to answer this.
Output only one line of JSON with left, top, right, bottom as 0-to-1 fractions of the image width and height.
Next: crop the original with those coordinates (plus 20% margin), upload it together with the overview image, ask the same question, and compare with the answer you got before step 1.Terminology
Related Papers
Gemini 4 Neon
Gemini 4 Neon cuts cost per task 32% versus Carbon, at $3 input and $15 output per million tokens
AI is killing All About Berlin
After Google AI Overviews, a Berlin guide site lost 70-75% of its traffic.
Not Everything Needs an LLM
PostHog used small decision models to skip 98.7% of expensive LLM calls
GPT‑6.2 Astra and Sol
GPT-6.2 Astra and Sol cut costs sharply, but over-engineering and early-stop reports are mixed
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
One trn2.48xlarge serves GLM-5.3-Flash through an OpenAI-compatible API using NKI megakernels.
Understanding Positional Embeddings in Modern LLMs
A walkthrough of how LLMs encode token position, from GPT-2 to RoPE
Show HN: DeepSeek V3 in ~200 lines of pure Jax
DeepSeek V3's core design in 190 lines of JAX, trained on a MacBook Air in 2.6 hours
Nvidia allegedly engineered scarcity by funding AI startups to buy its GPUs
WSJ alleges NVIDIA funded startups that then bought its own GPUs to prop up demand.
Related Resources
Original Abstract (Expand)
Online vision-language models often miss small details in large images, because the service shrinks the image, or rejects it, before the model ever sees it. A car 20 pixels wide in an 8000 by 6000 drone photo reaches Claude about 6 pixels wide, and OpenAI's default rejects the photo outright. We measure how much detail survives for seven models from four vendors, using an eye exam of printed codes and 601 questions about small objects in real photos. The smallest text a model reads half the time grows from 6 pixels or less on a small image to a predicted 10 to 13 pixels on a 7680 by 4320 photo at default settings. Zooming wins much of the detail back. Splitting each photo into up to nine tiles raises accuracy by 11.1 to 14.5 percentage points for Claude and Gemini, and by avoiding rejections it lifts GPT-6.1 Sol from 50.7% to 83.7%. A two-step zoom, in which the model first names the region it needs, matches or beats tiles on Gemini and comes within 2.5 points of them on Claude, with about 40% of their image tokens. When a detail is lost, Claude and Gemini both say they cannot tell about half the time, but they fail differently in the rest: Gemini gives a confident wrong answer in 43% to 48% of these cases, Claude in only 5% to 9%. A check based on detail size, run before the call, misses most of the gain from tiles on Claude and Gemini, but Claude's own reply is a reliable trigger: tiling only after an unsure reply comes within 1.4 points of always tiling with 62% to 70% of the image tokens. We turn these findings into five rules for when an agent should zoom, and release the eye exam, every model answer and code at https://siddvoh.com/know-when-to-zoom.