Faster Retrieval for Better, More Efficient RAG
TL;DR Highlight
Small neural nets learned query routing, hitting the same recall up to 8x faster in IVF search
Who Should Read
A backend developer serving millions of documents via vector search (IVF) and struggling to balance retrieval latency and recall in RAG
Core Mechanics
- Known query distributions can be learned
- Existing approximate search builds indexes from the key (document vector) structure only.
- In RAG, queries are short questions while keys are long documents, so they look very different.
- Production query distributions are stable, but generic methods leave that signal unused.
- MIPS is a support function problem
- The function mapping a query to its largest inner product with any key is the support function (a convex function).
- Its gradient returns the best-matching key itself.
- So a small neural network can be trained to mimic it via regression.
- SupportNet replaces cluster routing
- Standard IVF routes a query to the nearest centroid and misses outlier keys.
- SupportNet scores each cluster by how good the best key inside it is for the query.
- On Natural Questions (~2.6M passages, 128 clusters), the small model hit 95% routing recall about 1.9x faster than centroid routing (1.74 ms vs 3.36 ms).
- KeyNet acts like a query rewriter
- It takes a query and predicts the likely best-matching key vector.
- You keep your existing IVF index and just feed it the rewritten query.
- With FAISS IVF, reaching 99% recall took up to ~8x less time with the largest model (0.23 ms vs 1.8 ms for the raw query).
- Better recall at the same cost, less scanning
- Scanning 0.001% of the database, raw queries got 86% recall versus 89–97% for KeyNet (96.7% for the largest).
- Reaching 99% recall needed ~11x fewer keys (0.009% vs 0.10% raw).
- Which one to pick depends on your goal
- Use KeyNet when inference speed is the bottleneck and approximating the best key is enough.
- Use SupportNet when you need calibrated scores or have tight compute at small model sizes.
- Very large systems may use a SupportNet router plus one KeyNet per cluster.
Community Reactions
- Experience that BM25 beat learned methods
- One commenter found that for up to a few million items, a BM25 index with an LLM search tool casting a wide net won every time.
- They argued learned methods mainly matter at hyperscale with hundreds of millions of items and top-5 relevance needs.
- Spotify has a similar learned retrieval approach
- A commenter pointed to Spotify's 'Parallel Fusion Router'.
- It uses a teacher LLM to train a smaller router LLM with an LLM-as-a-judge reward.
- Pushback that grep beats embeddings on codebases
- A commenter who built RAG over codebases said embedding models never beat grep.
- They noted test files and docs pollute matches and need pruning.
- They suggested using an LLM subagent to scout instead of an embedding model.
- Opinion that ranking biases the LLM
- One commenter said heuristic retrieval biases LLMs, and letting the LLM explore on its own gave better final quality in their experience.
How to Apply
- Measure your recall baseline on real query logs
- Pull a few hundred real user queries and store exhaustive-search results as ground truth.
- Build a table of recall and latency for each IVF nprobe (clusters probed) setting.
- Use this table to decide whether learned routing is worth it.
- Check that your query distribution is stable
- This approach needs a known query distribution, so check recent logs for topic drift.
- If it shifts often, postpone adoption since the source says it is only robust to mild shifts.
- Try a query rewriter in front of IVF
- Train a small regression model on (query, best key) pairs and put it before your FAISS IVF.
- Compare total latency including the mapping step and recall against raw queries.
- Try BM25 plus an LLM search tool first if you're small
- If you have a few million documents or fewer, wire a BM25 index as an LLM search tool and compare on the same query sample.
Try It
# Paste into Claude or ChatGPT (simplified version of the paper's method: judge fit via query log analysis, no training)
I run a RAG retrieval system. Based on the info below, judge whether 'query-distribution-learned routing (SupportNet/KeyNet style)' fits us.
- Number of documents (keys): {{doc count}}
- Vector dimension: {{dimension}}
- Current index: {{e.g. FAISS IVF, 128 clusters}}
- Current latency and recall: {{numbers}}
- 20 recent user query samples: {{query list}}
Answer the following.
1. Assess how narrow and stable the query topics are
2. Design a recall experiment using exhaustive search as ground truth (sample size, metrics)
3. Give criteria for whether BM25 + an LLM search tool might be better if we have a few million documents or fewer
Next step: run the designed experiment on part of your query log and paste the result table back for a verdict.Terminology
Related Papers
FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs
Adding 'verify and prioritize the passage' to RAG prompts raised false-passage adoption; a knowledge-priority warning fixed it.
Memorilla: Latent Semantic Memory for LLMs
Compressing documents into 16 memory tokens beat full-context accuracy on FactKG.
What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion
Preserving red text and highlights in docx ingestion lifted marked-requirement F1 from 0.150 to 0.690.
Multimodal Reranking Is a System One Decision
One multiple-choice question to a VLM over 16 candidates reranks them, beating a dedicated 2B reranker.
You could have invented transformers from memory
Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million
Pointing AI at archives found a forgotten meteorite, lost rhinos, and more
Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives
EmbeddingGemma 2: An open, lightweight multimodal embedding model
One open 740M model embeds text, images, audio and video into a single shared vector space
RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation
Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.