What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

Who Should Read

Backend developers feeding Word or PDF policy documents to RAG or agents who see key clauses missed or wrongly flagged.

Core Mechanics

• Converters silently drop color
  • Docling, plain text and other converters keep text and tables but lose font color and highlights.
  • The pipeline looks healthy and no monitor fires.
• Agents guess when they can't see the cue
  • Docling's false-positive rate on unmarked requirements is 0.043, about four times this paper's 0.011.
  • Some models go the other way, with recall 0.09 and precision 0.96.
• The loss sits in highlight and color
  • Removing one cue at a time cost +0.302 macro F1 for highlight and +0.262 for color.
  • Bold, italic, monospace and strikethrough each stayed within ±0.003 of zero.
• Bold fails because it is too common
  • One document can carry over a thousand bold markers, so bold says nothing about intent.
  • Color and highlight are rare and deliberate, so they carry signal.
• Preserving isn't enough, add meaning
  • Telling the agent a span is a critical warning beat showing raw [COLOR:#FF0000] by +0.085 macro F1.
  • The p-value is borderline (0.049), and two sparse-answering models like Sonnet 5 showed no benefit.
• The cost is roughly 2x tokens
  • Cue recovery is code-only with no extra LLM calls.
  • Tokens are about twice plain text, trading context budget for accuracy.

Evidence

• Formatting-blind conversion collapsed F1
  • On 76 real policy documents micro F1 was 0.690 for this method, 0.586 docx2python, 0.150 docling and 0.075 plain text.
  • Across six models (five Claude, GPT-5.6) plain text scored 0.007–0.075 every time.
• Leave-one-cue-out isolated the cause
  • Removing color dropped color-marked documents from 0.699 to 0.214 micro F1; removing highlight dropped highlight documents from 0.659 to 0.128.
  • Documents using the other cue moved by at most 0.011.
• Typing the cue helped in a pre-registered test
  • On 79 documents with 6 repeats, typed scored 0.667, untyped 0.597 and docx2python 0.570.
  • Typed beat untyped by +0.085 (pHolm 0.049) and docx2python by +0.107 (pHolm 0.014).
• Microsoft OCB and a full pipeline agreed
  • On 326 Word File-Fidelity items the mean was 0.959 for this method versus 0.810 docx2python, 0.720 docling and 0.634 plain text.
  • In the identical full pipeline on 68 documents micro F1 was 0.503 versus 0.258 docx2python and 0.084 docling.
• The problem is common in real files
  • 32.8% of 6,462 docx files carry deliberate red text or highlight.
  • 10.3% carry one on an obligation sentence.

How to Apply

• Count color and highlight in your corpus
  • Today, run a python-docx script counting runs with font color or highlight_color.
  • Check how many obligation sentences carry them to decide priority.
• Keep formatting tags in converted text
  • Preserve color and highlight spans as [COLOR:..] or [HIGHLIGHT:..] tags when converting docx.
  • Keep bold, italic and strikethrough as Markdown and serialize color and highlight as custom tags.
• Attach semantic roles to the tags
  • Map red text or red highlight to CRITICAL_WARNING, other highlight to WARNING and strikethrough to DEPRECATED.
  • Drop roles for common cues like bold and keep only the high-signal ones.
• Add assertions at ingestion
  • Add an assert in the parser that a recovered cue implies a located text span.
  • Output monitoring won't catch this loss, so check at ingestion.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: reading a document with formatting tags)

In the document below, [COLOR:..] and [HIGHLIGHT:..] tags mark text the author deliberately emphasized.
Treat red ([COLOR:#FF0000]) as CRITICAL_WARNING, any other highlight as WARNING, and ~~strikethrough~~ as DEPRECATED (no longer valid).
Do not treat bold as an emphasis signal.

{{document: text with formatting converted to tags}}

Question: List only the requirements in this document that the author singled out for special attention.
Do not include unmarked requirements.

Next to try: ask the same question on a plain-text version with all tags removed and compare the answers.

Terminology

Related Papers

• Multimodal Reranking Is a System One Decision

  One multiple-choice question to a VLM over 16 candidates reranks them, beating a dedicated 2B reranker.

• You could have invented transformers from memory

  Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

• Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

  Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

• EmbeddingGemma 2: An open, lightweight multimodal embedding model

  One open 740M model embeds text, images, audio and video into a single shared vector space

• RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

  Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.

• Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

  Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.

• Microsoft exec called AI scraping 'the largest theft of labor in human history'

  NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping

• Harnessing the Universal Geometry of Embeddings

  Translating embeddings between models without paired data exposes vector database security risks