Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows

Who Should Read

Backend developers building pipelines where an LLM reads long documents and then makes a judgment or recommendation

Core Mechanics

• Found it, but didn't use it
  • Adding only unrelated text from 2,000 to 128,000 tokens pushed the disclosure's influence on judgment to the noise floor.
  • Direct factual questions on the same documents stayed accurate at every length.
• Order survives, magnitude doesn't
  • The judgment range across a seven-step severity ladder shrank by a factor of 5.6.
  • Pairwise ordering accuracy fell much less, from 0.93 to 0.79.
• Bigger models only delay it
  • The 27B model kept an effect at 128k, roughly what the 9B models achieve at 2k.
  • The effective decision context is shorter than the advertised window.
• Chunk-and-summarize evicts the fact
  • Budgeted notes recorded revenue and expenses from the head of each segment and crowded out the risk paragraph.
  • Influence was zero even at 2k, where the document fits in one chunk.
• A targeted answer right before the decision works

Evidence

• Influence decline for Qwen3.5-9B
  • Sell-probability effect was +0.032 at 2k, +0.005 at 8k, +0.001 at 32k, and +0.004 at 128k.
  • At 32k and 128k it was indistinguishable from irrelevant-paragraph insertion noise (p>0.5).
• Retrieval stayed intact
  • The 9B model retrieved the disclosure for all 12 firms at 128k.
  • The production model's retrieval completeness was 0.77, 0.76, 0.72, 0.74 across the four lengths.
• Workflow comparison
  • Retention of short-context influence at 128k was 12% for baseline and 67% for extract-then-decide.
  • Baseline went from +0.032 to +0.004; extract-then-decide from +0.126 to +0.085.
• Chunk summaries and real filings
  • The notes audit found the disclosure absent in 24 of 24 cells at 2k and 21 of 24 at 128k.
  • On 12 real 10-Ks, extract-then-decide reached +0.043 with 11 of 12 firms positive.

How to Apply

• Add a targeted extraction step before the final judgment prompt
  • In an existing long-document judgment prompt, first ask about the key matter (e.g., covenant terms or settlement amount).
  • Put that answer right before the decision question and keep the original document in context.
• Evaluate by judgment change, not retrieval accuracy
  • Run the document with the key paragraph and a version where it is replaced by equal-length neutral text, then compare outputs.
  • If accuracy is high but the judgment doesn't move, treat the fact as not integrated.
• Audit your generic chunk-and-summarize pipeline
  • Save intermediate notes and check whether key facts survive.
  • If they don't, replace generic summaries with targeted per-issue extraction.
• Don't rely on reasoning effort alone
  • When judgments don't move on long documents, change information placement and form before raising effort.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: answer a targeted question before judging)

[Document]
{{long document}}

[Step 1] From the document above, summarize in a structured way what it says about {{key matter, e.g., loan covenant terms}}, including amounts and current status.

[Step 2] Using the summary you just wrote, and consulting the original document again, answer {{decision question, e.g., what is the 12-month default probability in %}}.

Next, try this: replace that paragraph with a neutral one, run the same procedure, and compare whether the answer changes.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.