Better Together: Complementary Query Rewriting Under a Strong RAG Baseline

Who Should Read

Backend developers who added query rewriting to RAG but are unsure the extra LLM calls are worth the cost

Core Mechanics

• Rewriting alone barely helps
  • On top of a strong pipeline with reranking and MMR, solo rewriting gains were small.
  • The best method changed by dataset, so no single one wins.
• Combining methods helps a lot
  • Methods fail on different questions, so they cover for each other.
  • On enterprise data, a four-method union reached HIT@10 of 51.70 vs 39.22 for the dense baseline, +12.5 points.
• It's not just a bigger search budget
  • Budget-matched controls (TopK-100 reranking, multi-pass HNSW) captured only 40-41% of the gain.
  • The remaining ~60% came from complementarity.
• Merging overlapping results hurts
  • On AmbigNQ the methods retrieve nearly the same documents (Jaccard@10 0.55-0.99).
  • Merging scored 2.4 points below the best solo method.
• Rewrite only when the score is low

Evidence

• Enterprise ensemble results
  • On EnterpriseRAG-Bench (470 queries), S1+S3+S4+HyDE reached HIT@10 51.70 vs 39.22 for S1, +12.5 points.
  • The five-method union hit 52.98 (+13.8) vs an oracle of 55.74 (seed 42).
• Budget-matched controls
  • On Enterprise, TopK-100 reranking and multi-pass HNSW both reached HIT@10 of about 44.3, +5 over S1.
  • That explains only 40-41% of the four-way gain of +12.5.
• Router simulation (τ=0.65)
  • Enterprise HIT@10 went from 39.22 (S1) to 43.55 with 39.0% of queries expanded; always-merge got 47.94.
  • AmbigNQ expanded only 7.2% and got 93.76 vs S1 93.67; always-merge dropped to 93.42.
• Answer quality
  • On 200 Enterprise queries with a GPT-4.1 reader, the router improved F1 by +1.92 over S1 (p=0.008).
  • Always-merge gained +2.46, and the gap between them (−0.54) was not significant.

How to Apply

• Check top-1 score distribution in your RAG logs today
  • Split queries into those with the gold doc in the top-10 and those without, and average the reranker top-1 score for each.
  • Use the midpoint as your threshold τ candidate (the paper used 0.65 from 0.711 and 0.612).
• Add Query2Doc only on low scores
  • If the top-1 score is below τ, generate a pseudo-document with an LLM, search again, and merge by per-document max score to take top-K.
  • If it's at or above τ, return the original results only.
• Measure result overlap before merging
  • Compute Jaccard between each method's top-10 lists.
  • If overlap is high (0.55 or above), don't merge and keep a single method.
  • Turn on ensembles only for low-overlap, enterprise-like corpora.
• Use S4 when you need cheap rewriting
  • Generate one comparison question between the subject and alternatives and run an extra search.
  • Pick it over Query2Doc when LLM call cost matters.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: generate 3 query rewrites)

From the user question below, create 3 search queries.
1) Query2Doc: a short pseudo-answer paragraph for the question
2) Synonym: the question reworded with different vocabulary (S3)
3) Comparative: a question comparing the subject with alternatives (S4)

Question: {{질문}}

Next step: run a search with each query, collect the top-10 document IDs, and compare their overlap with the results of the original query.

Terminology

Related Papers

• You could have invented transformers from memory

  Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

• Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

  Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

• EmbeddingGemma 2: An open, lightweight multimodal embedding model

  One open 740M model embeds text, images, audio and video into a single shared vector space

• RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

  Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.

• Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

  Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.

• Microsoft exec called AI scraping 'the largest theft of labor in human history'

  NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping

Related Resources

• https://huggingface.co/datasets/onyx-dot-app/EnterpriseRAG-Bench