Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out
The Armature team conducted a sprawling experiment involving nearly 17,000 runs to scrutinize how leading coding agents like Claude Code, Codex, and Cursor select third-party developer tools. This deep dive reveals the nuanced decision-making processes of AI in coding, from source utilization to contextual influences. It's a critical investigation for developers and vendors alike, as AI agents rapidly become central to the software development workflow.
The Lowdown
The Armature team undertook an ambitious research project to understand the preferences and decision-making mechanisms of prominent coding agents when choosing third-party developer tools. This study, prompted by the increasing role of AI in coding workflows and its implications for both developers and tool vendors, involved a massive dataset to uncover how agents discover, evaluate, and ultimately select solutions for specific programming tasks.
- Motivation: To analyze agent tool selection, vital for developers to trust AI judgments and for vendors to ensure their products are chosen by these increasingly influential coding assistants. The study noted Vercel's observation that over 30% of deployments were agent-initiated, a 1000% increase in six months.
- Experiment Scale: Measured 16,893 sessions across 51 codebases and 18 sectors, involving 1,163 prompt variations and 75 repositories, with Claude Code, Codex, and Cursor implementing solutions rather than just recommending them.
- Methodology: Employed diverse personas (vibe-coder to enterprise engineer), real-world tasks, and ephemeral sandboxes. A "simulated human" (Gemini 3.7 Flash) orchestrated interactions to mimic realistic development conversations, and another Gemini instance judged session validity and tool choices.
- Key Learnings:
- Agent Divergence: Agents utilize different information sources (web search vs. priors) and frequently disagree on tool choices, agreeing in only 42% of cases. Claude Code, for example, builds in-house almost twice as often as Codex and Cursor.
- Context is King: Repository context, such as programming language or framework, heavily influences tool selection, leading to different winners for the same task in different environments.
- Mention ≠ Win: Many widely known tools are frequently mentioned by agents but rarely selected for implementation (e.g., PayPal, LangChain, Supabase).
- Vendor Page Impact: Specific details or how information is presented on vendor websites (e.g., Mailgun's retention policy, Supabase's bundled features) can significantly sway agent decisions.
- Market Dynamics: Some tool categories exhibit strong dominance by a few players (e.g., Stripe, Neon, Amazon S3), while others are more contested.
Armature plans to release further insights from this extensive dataset, which is publicly available for examination. This foundational research offers a transparent look into the evolving intelligence of coding agents, providing invaluable data for the future of software development.