HN
Today

Grep beats LSP? Why coding agents ignore your fancier tools

This technical dive challenges the intuition that more precise tools are always better for AI agents, revealing why LLMs often prefer humble grep over sophisticated Language Server Protocols (LSP) for code tasks. It highlights that an agent's 'harness'—the runtime and tool interface—is as crucial as the model itself. The findings offer practical lessons for engineers building AI coding assistants, emphasizing context, output format, and task-specific routing.

17
Score
1
Comments
#6
Highest Rank
2h
on Front Page
First Seen
Sep 4, 4:00 AM
Last Seen
Sep 4, 5:00 AM
Rank Over Time
76

The Lowdown

A study investigated why large language model (LLM) coding agents frequently favor basic lexical search tools like grep over more advanced, semantically precise tools like those backed by Language Server Protocols (LSP) for code retrieval. The surprising results suggest that 'LLM-friendliness'—how well a tool's output integrates into the model's workflow—is often more important than raw semantic precision.

  • LLM agents showed a strong preference for grep in simple code localization and renaming tasks, choosing LSP only 0-6% of the time when both were available.
  • Forcing agents to use LSP first in some cases reduced task success, indicating that precision alone doesn't guarantee utility.
  • The study found that task type significantly influenced tool choice: agents used LSP 45-57% of the time for 'reference-completeness' tasks (e.g., finding all callers), where semantic precision provided a clear advantage.
  • Codebase 'noise' (lexical ambiguity, not static typing) was a key factor; LSP offered substantial F1 score gains in noisy repositories but no benefit in clean ones where grep already performed well.
  • A crucial finding was that changing LSP's output format to include inline source context (similar to grep's output) dramatically improved agent success rates and reduced follow-up file reads, demonstrating the importance of how information is presented to the model.
  • grep retains a structural advantage for tasks requiring text-wide completeness, such as updating comments or configuration files, which semantic tools are not designed to handle.
  • The article introduces the concept of a 'harness' (the runtime, instructions, tool schemas, output shapes) as integral to an agent's capability, stating that 'agent capability = model × harness'.
  • The author's team developed AgentConnect, a platform that uses the open Agent Client Protocol (ACP) to allow native coding agent runtimes to operate within their familiar tool loops, recognizing that each model's effective capability is tied to its specific environment.

The research underscores that the true efficacy of a retrieval tool for LLM agents cannot be evaluated in isolation. Instead, it's a function of the entire agent system, where factors like output format, task alignment, and the agent's learned workflow patterns dictate whether a tool, even a semantically 'superior' one, will be adopted and used effectively.