What's the best programming language for coding agents?
Dan Luu dives deep into the question of which programming languages best suit LLM coding agents, challenging the popular notion that dynamic languages are inherently more token-efficient. Through custom evaluations of complex tasks like Zstd decoders and a board game, he reveals that previously strong claims about language efficiency often don't generalize beyond trivial problems, emphasizing that real-world LLM coding performance is far more nuanced. Hacker News revels in the empirical debunking and speculates on the true drivers of LLM code quality and the peculiar challenges of evaluating AI agents.
The Lowdown
A widely cited idea suggests dynamic languages are more token-efficient for LLMs due to their conciseness, sometimes by a factor of 2-3x compared to static languages. Dan Luu's post scrutinizes this claim, arguing that such conclusions often stem from evaluations using trivial problems and that issues in existing benchmarks can skew results.
Luu conducts his own evaluations to build intuition, pre-registering several hypotheses:
- Challenging Dynamic vs. Static Claim: He predicted the claim that dynamic languages are superior wouldn't hold for larger, more complex problems, drawing parallels to how trivial benchmarks often mislead. His evaluations ultimately supported this, finding mixed results or advantages for static languages at higher effort levels.
- "Weird" Language Supremacy: The idea that extremely dense languages like J would be superior was also dismissed, with Luu predicting AI labs would have less training data for obscure languages. This also largely held true.
- Popularity Correlation: While not a strong claim, his data weakly supported that more popular languages yielded more correct and cheaper solutions.
His core experiments involved two distinct, non-trivial tasks:
- Zstd Decoder Implementation: LLMs were tasked with implementing a zstd decoder from its RFC. While dynamic languages initially appeared more efficient at 'medium' effort, at 'ultra' effort, static languages showed comparable or even better results. The strong relationships observed in trivial evals did not generalize.
- Pandoc ProgramBench Adaptation: This TDD-like task involved modifying the ProgramBench eval. Again, no strong correlation was found between success/cost and language type. Obscure languages generally performed poorly, though Clojure surprisingly excelled here.
He also attempted a third eval, implementing the rules for the board game "Guards of Atlantis 2," which involved ambiguous, contradictory, and "common sense" interpretation—tasks where LLMs universally scored near zero, indicating a significant challenge for current models in navigating human-like, unclear specifications.
Luu concludes that many existing claims about language suitability for LLMs are likely incorrect, but determining what is truly effective requires extensive, diverse evaluations due to high variance across tasks. He highlights that while LLMs make such empirical studies feasible (despite their cost in tokens), the incentives for AI labs to publish detailed findings are not strong, leaving many questions unanswered.
The Gossip
Evaluation Exclusions & Efficacy
Commenters debated the practical implications and validity of the evaluation's setup. Many questioned the decision to air-gap LLM agents, arguing that real-world agents use search and access external resources, making the experiment 'unnatural'. There was also skepticism about whether replicating existing, well-known software truly serves as a useful signal for LLM capabilities, with some suggesting that models might simply be retrieving and re-styling code from their training data rather than 'reasoning' from scratch.
Language Traits & LLM Performance
Discussion frequently revolved around specific language characteristics and their impact on LLM performance. Go was often lauded for its consistency and clear idioms, making it easy for LLMs to generate reliable code. Conversely, Python's perceived 'pollution' by varied coding styles was noted as a potential challenge. The debate between static and dynamic typing continued, with some arguing that static types (like Rust's) offer faster verification loops at compile time, reducing LLM iteration cycles, while others pointed to Python and JavaScript's surprisingly strong performance in the article's benchmarks despite their dynamic nature.
The Codebase Context & Quality
Many commenters explored how the surrounding codebase and the nature of the training data influence LLM output. It was suggested that a well-designed, componentized Python application might be more LLM-friendly than a monolithic Rust project, as LLMs could focus on smaller, verifiable contracts. There was also a recurring observation that esoteric languages with smaller, high-quality enthusiast communities might yield better LLM results due to less 'noisy' training data, whereas popular languages like Python, with vast amounts of beginner or poorly structured code, could present greater challenges.
Meta: Dan Luu's Blog Design
A significant portion of the comments veered into a meta-discussion about Dan Luu's signature minimalist blog styling. Some expressed frustration with the lack of basic CSS, finding the text dense and hard to read, and questioned why an author known for technical excellence would neglect such a fundamental aspect of presentation. Others defended it as an intentional artistic or engineering choice, perhaps to demonstrate web bloat minimization, and suggested users leverage browser 'reader mode' or custom stylesheets, leading to a humorous debate comparing it to being served raw ingredients at a restaurant.