Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

Who Should Read

Backend developers building document QA over PDFs or internal knowledge bases who must choose between OCR text and page images as LLM input.

Core Mechanics

• Images win on accuracy every time
  • On documents of 50 pages or fewer, images beat text at every length bucket.
  • This held on both the enterprise KB and MMLongBench-Doc.
• Images get slower as documents grow
  • Text latency stayed roughly flat regardless of document length.
  • Image latency kept rising with length.
• Text and images miss different questions
  • Exactly one representation was correct on 19-25% of items.
  • Neither one fully replaces the other.
• Text+image depends on the model
  • GPT models gained about 5 pp over image-only on the KB.
  • Gemini Pro's text+image fell below image-only on MMLongBench-Doc FULL.
• A question-only router helped
  • A TF-IDF logistic regression chose text or image per question.

Evidence

• Image vs text accuracy (ORACLE)
  • Gains over text reached up to 14.8 pp on the KB and 11.6 pp on MMLongBench-Doc.
  • KB average accuracy: text 0.611, image 0.708, text+image 0.731.
• Cost and latency (averaged over 4 models, 2 datasets)
  • Under ORACLE, text to image raised cost 0.61 to 0.91 $/100q (1.5x) and latency 3.3 to 4.9 s.
  • Under FULL, image cost 1.9x and text+image 2.4x of text; image latency was 7.0 s vs 3.7 s for text.
• Router results (FULL, document-disjoint held-out)
  • Overall accuracy: always-text 63.6%, always-vision 67.7%, router 66.2%.
  • Median latency fell from 4.4 s (always-vision) to 3.1 s, about 30%.
• Router gain by dataset
  • KB rose from 65.0% to 67.8% (+2.7 pp).
  • MMLongBench-Doc rose from 53.9% to 55.9% (+2.0 pp).

How to Apply

• Compare text vs image on 50 of your own samples
  • Run 50 real questions with OCR text and with page images, and record accuracy and latency.
  • Keep the prompt and grading identical; change only the input format.
• Find the question types where they disagree
  • Collect questions only one format got right and tag them by table, screenshot, or diagram reference.
  • Use the image-needed types as candidate routing rules.
• Add a question-text router
  • Train TF-IDF plus logistic regression (class_weight=balanced) on the two-format labels.
  • Send only questions predicted to favor images to page-image input.
• Check your image cap against document length
  • Find where your provider's page limit (50 in the paper) truncates documents.
  • Default to the text path for documents beyond the cap.

Try It

# Paste into Claude/ChatGPT (simplified version of the paper's method: decide if a question needs images)

You are a document QA router. Looking only at the question below, decide whether answering needs page images (tables, charts, screenshots, layout) or whether OCR text is enough.

Question: {{question}}

Output format (JSON): {"route": "text" or "image", "reason": "one sentence"}

Next step: run 20 questions through this, then compare against actual results from running text and image inputs.

Terminology

Related Papers

• You could have invented transformers from memory

  Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

• Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

  Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

• EmbeddingGemma 2: An open, lightweight multimodal embedding model

  One open 740M model embeds text, images, audio and video into a single shared vector space

• RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

  Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.

• Microsoft exec called AI scraping 'the largest theft of labor in human history'

  NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping

• Harnessing the Universal Geometry of Embeddings

  Translating embeddings between models without paired data exposes vector database security risks