Grep beats LSP? Why coding agents ignore your fancier tools

TL;DR Highlight

Adding two lines of source to LSP results lifted rename pass@1 from 0.67 to 0.83.

Who Should Read

A developer who added LSP or MCP code-navigation tools to a coding agent and finds the agent barely uses them

Core Mechanics

Agents default to grep
- On simple localization tasks, three models picked the semantic (LSP) tool only 0–6% of the time when both were available.
- Forcing a semantic-first path cut success from 100% to 89%.
Reference tasks split about 50/50
- On reference-completeness tasks, where the agent must find every caller, models chose semantic navigation 45–57% of the time.
- Precision was 1.00 for LSP versus 0.76 for grep, but recall stayed near 0.66 in both.
LSP pays off only in noisy repos
- On remeda, grep precision was 1.00, so LSP added no F1 and used 16% more tokens.
- On hono, grep precision was 0.51, so LSP gave +0.246 F1 and used 12% fewer tokens.
- Both are TypeScript with opposite results, so grep's false-match rate predicts the outcome better than static typing.
Two lines of source changed everything
- The LSP tools first returned only file, line, and column, so the agent had to open files to see the code.
- Attaching two lines of source on each side raised multi-file rename pass@1 from 0.67 to 0.83.

Community Reactions

Asking the agent what tool would have helped
- One commenter watches Claude Code work, then asks which tools would have made the job easier, which led them to fzf and others.
- They built a Python CLI that converts data to Parquet for DuckDB queries and cut analysis turnaround.
Sparse AST printing worked well
- One commenter reported that a skill printing types and function signatures without bodies cut exploration time and tokens significantly.
- They said the same tool's editing feature worked poorly because models are tilted toward common editing tools from post-training.
Doubts about the experimental setup
- Some commenters said the author used their own harness and should also test Claude Code, which has direct LSP support.
- Others noted the study measured token economy, while quality and difficulty matter more to people not billed per token.
- One pointed to open Claude Code LSP/pyright issues that may make grep look more stable.
Ways to get the LSP actually used
- One commenter said a SKILL instructing the agent to prefer LSP worked well.

How to Apply

Count whether the agent calls the new tool
- Today, count LSP/MCP tool calls versus grep calls per task type in your agent logs.
- If the call rate is near zero despite availability, fix the tool description and result shape first.
Return path:line:content with context
- For custom tools or MCP servers that return bare locations, attach two lines of source on each side of every result.
- Compare follow-up file reads and success rate before and after.
Route search tools by repo noise
- Run grep on your common identifiers and check the false-match rate first.
- Enable semantic navigation only on noisy repos, and keep grep alone on clean ones.
Keep a grep fallback and compare at equal accuracy
- Leave a grep path open for tasks like rename that must touch comments and strings.
- Compare token use only when success rates match.

Try It

copy & run

# Paste into Claude Code, Claude, or ChatGPT (simplified version of the paper's method: audit tool output shape)

I'm building a tool for a coding agent. Below is an example of the response my tool returns to the agent.

{{example tool response (JSON or text)}}

Audit this response against these criteria.
1. Can the agent judge each result's relevance without opening the file separately?
2. What would it look like in grep-style path:line:content form? Show an improved version with two lines of source on each side.
3. What does this tool miss (comments, strings, config files), and what should the tool description say so the agent falls back to grep?

Next step: run the same 5 tasks with the improved output and compare follow-up file reads before and after.

Terminology

LSP (Language Server Protocol)A standard protocol where editors ask a language server for code-aware features like go-to-definition and find-references. It navigates by code structure, not text matching.

HarnessEverything around the model: instructions, tools, input schemas, result shapes, and the loop. The same engine drives differently in a different chassis.

pass@1The share of tasks solved on the first attempt.

F1A score combining precision (share of returned results that are correct) and recall (share of true results found). No change means retrieval quality did not move.

RecallThe share of true call sites the agent eventually found. Cleaner tool output does not raise it if the agent stops early.