The AI cost of JavaScript frameworks
Nolan Lawson challenges web development's 'vibe-based' framework choices, using AI agents to benchmark JavaScript frameworks across cost, quality, and maintainability. This deep dive provides concrete data on how AI builds with popular tools like Next.js, Astro, and SvelteKit, sparking debate on the future of dev decisions. It scrutinizes whether framework popularity aligns with tangible benefits in an AI-driven coding era, pushing developers to reconsider their 'npm install' choices.
The Lowdown
Nolan Lawson takes aim at the prevailing 'vibe-based' decision-making in web development, arguing that even with the advent of AI coding agents, technical choices are often made without evidence. His article highlights the shift from manual coding to an 'agentic era,' where developers are less constrained by developer experience (DX), yet still default to popular frameworks without rigorous evaluation. To counter this, Lawson conducted a benchmark, employing AI agents to build a Pokémon application across various JavaScript frameworks.
- Methodology: Lawson tasked Claude Opus 5.5 AI agents to build a small, non-trivial Pokémon app using 12 different JavaScript frameworks/meta-frameworks (e.g., Next.js, Astro, Hono+htmx, Angular, SvelteKit, Vanilla JS). Each framework was tested through five iterations, measuring development cost, quality, and maintainability.
- Cost Analysis: Measured by API prices (tokens, time), Hono+htmx, Astro, and Nuxt emerged as the most cost-effective, while SvelteKit, Gracile+Lit, and Angular were the most expensive. The variance highlighted the stochastic nature of LLMs and inconsistent AI tool (MCP/skill) usage across frameworks.
- Quality Metrics: Evaluated using Lighthouse scores (Performance, Accessibility, CLS) for the homepage. Most frameworks achieved near-perfect scores for the simple app. More granular metrics like JavaScript bytes transferred, Total Blocking Time (TBT), and Largest Contentful Paint (LCP) showed clearer differentiation, generally favoring Multi-Page Applications (MPAs) over Single-Page Applications (SPAs).
- Maintainability (Lines of Code): Measured by non-blank source file lines, Hono+htmx, Nuxt, and Astro produced the most succinct codebases. Angular resulted in the highest lines of code, impacting overall cost and illustrating Rich Harris's idea that frameworks should organize the mind, not just code.
Lawson concludes by advocating for evidence-based decision-making in framework selection, especially as AI agents become integral to development workflows. He stresses that his benchmark is not definitive but aims to inspire further scrutiny and improvement in the web development community, pushing for better, faster, and more accessible web applications.
The Gossip
Benchmark's Breadth and Bias
Many commenters debated the representativeness and scope of the benchmark. Some found the Pokémon app too simplistic to draw broad conclusions for complex, interactive applications, arguing that larger projects would highlight different framework strengths. Others defended the benchmark's value, acknowledging its stated limitations but appreciating the author's scientific approach to 'vibe-coding.' The author's own caveats were noted, with calls for future benchmarks to include more complex features like authentication or real-time interactivity.
Framework Fanfare and Failures
Users reacted strongly to specific framework performance. Astro, Hono+htmx, and Nuxt received praise for their strong showings, especially Astro's minimal JavaScript output. Angular, conversely, was a frequent target, with some commenters expressing satisfaction at its poor performance and high cost, while others defended it as suitable for large, complex enterprise applications, arguing its rigidity is a feature, not a bug. There was also discussion about the impact of SvelteKit's recent V3 migration guide skewing its results due to AI agent behavior.
The Agentic Era: AI's Impact on Dev
A core theme revolved around the 'agentic era' and AI's role in coding. Some users expressed apprehension about AI taking over manual coding, fearing a loss of human skill, with one commenter hoping to be the 'last person on Earth writing code by hand.' Others speculated on how AI might fundamentally change framework selection, suggesting that the cost of tokens and the ability of agents to refactor code might make framework structure less critical, or conversely, make frameworks even more essential for organizing AI-generated code.
Vanilla vs. Vibe-Driven Bloat
A significant portion of the discussion echoed the author's critique of framework bloat, with many advocating for vanilla JavaScript, plain HTML, or 'sprinkles of JS' with server-side rendering (HTMX). Proponents argued these approaches offer better performance, fewer dependencies, and simplified maintenance. Detractors countered that for truly complex, interactive web applications, frameworks are necessary to manage complexity and provide structure, preventing 'vanilla' projects from devolving into unmaintainable messes.
Framework's Fundamental Function
Commenters delved into the underlying purpose of JavaScript frameworks. Some asserted that frameworks primarily serve corporate interests by easing developer hiring and reducing training costs, rather than genuinely empowering individual developers. Others argued frameworks abstract architectural complexity, allowing developers to focus on business logic. The debate touched on the 'invented here' syndrome in large organizations and the trade-off between framework-imposed rigidity and developer flexibility.