RAG Is Simpler Than You Think

TL;DR Highlight

Start RAG with BM25 and query rewriting; add embeddings only when data shows you need them.

Who Should Read

A backend developer building RAG who is unsure whether to jump straight to a vector DB and reranking.

Core Mechanics

Jumping to embeddings overcomplicates things
- The author says many teams go straight to embeddings, vector DBs and reranking.
- Users often just want to find the doc on resetting a password.
Recipe 1 is plain BM25 full-text search
- With Elasticsearch or Postgres full-text search, API cost is zero and latency is under 10ms.
- It needs no chunking or eval setup and has no model deprecation risk.
- It misses synonyms ("car" vs "automobile") and struggles with semantic questions ("How do I...?").
Query rewriting is easier to fix than embeddings
- An LLM turns messy questions into keyword queries, about $0.001 per query with GPT-4o-mini.
- If results are poor, you edit the system prompt instead of re-embedding.
- Put internal terms (e.g. Atlas) in the prompt so they are kept as exact keywords.
Pick the embedding approach by data profile
- High churn favors on-the-fly embedding, which adds 200-500ms per query.

Community Reactions

Practitioner experience backing BM25-first
- Someone with large-scale RAG experience says full-text search is underrated and embeddings overrated.
- They say embeddings lead to re-embedding, reranking and vector search operations burden.
Opinion: agentic query rewrite on Lucene is the end game
- The commenter says letting the agent query the store iteratively makes capabilities unbounded.
- They argue embeddings stack non-determinism on non-determinism, while lexical search is easier to debug.
Experience dropping indexing from a coding agent
- Retrieval returned zero results due to a bug for a while and nobody noticed, and things worked better.
- After A/B tests they dropped indexing and rely on grep, though very large codebases may differ.
Pushback and criticism of the article
- One commenter argues 90% of document RAG projects should use embedding semantic search as the primary method.
- Others say the article reads as AI-written and lists whats without whys.
- Some found the on-the-fly and sparse+dense reranking explanations unclear.

How to Apply

Audit your search failures by query type
- Within 30 minutes, collect 20 recent failed queries and label them keyword, conversational or internal-term.
- Run the prompt below and eyeball the rewritten queries.
Put your internal glossary in the rewrite prompt
- List internal framework and service names as do-not-modify keywords in the system prompt.
- Feed rewritten queries to your existing BM25 or Postgres full-text search and compare.
Run a 2-week A/B test before adding embeddings
- Compare a rewrite group against your current search for two weeks.
- Add hybrid (BM25 top 50-100 reranked by embeddings) only if data shows it is needed.
Decompose multi-intent queries
- Have the LLM output sub-queries as a JSON list when a question has several intents.
- Call BM25 per sub-query in parallel and merge the results.

Try It

copy & run

# Paste into Claude or ChatGPT (the article's query-rewriting method as a prompt)

You are a query rewriter for document search. Turn the user's conversational question into BM25 keyword queries.
Rules:
- Remove stopwords (filler like "how do I").
- Add synonyms.
- Split multi-intent questions into several sub-queries and output a JSON array.
- Never modify the internal terms below; use them as exact keywords.
{{internal terms: name - description list}}

Question: {{user question}}

Output: only the JSON list of queries.

Next step: run your 20 failed queries through this, feed the results to your BM25, and check whether results improve.

Terminology

BM25A classic search algorithm that scores documents by how often and how distinctively keywords appear. Think of it as a smarter Ctrl+F.

Query rewritingAn LLM step that translates a user's phrasing into search terms the engine handles well.

Hybrid searchUse keyword search to pick candidates, then reorder them with embeddings.

On-the-fly embeddingEmbed only the candidate documents at query time instead of storing vectors. No storage is needed but each query pays latency.

Hot/cold tiersPre-embed frequently accessed docs and embed rare ones on demand. Like stocking only the popular items.