Astra for Coding: Why Are We Doing This Again?
The author critiques OpenAI's Astra for its "slop" code, excessive costs, and strange coding practices, suggesting AI development has entered an "involutionary" cycle where intense effort doesn't yield human-usable improvements. This piece resonates deeply with developers grappling with the practical frustrations and hidden costs of cutting-edge AI tools. It sparks a broader debate on whether current AI trajectories genuinely serve software engineering or are merely optimized for misleading metrics.
The Lowdown
Armin Ronacher, a core developer for CPython, expresses profound skepticism and frustration with OpenAI's latest model, Astra, for software engineering tasks. He introduces the concept of "Neijuan" (involution) to describe a system that demands increasing effort and competition without improving output, believing this applies directly to the current state of AI development.
- Ronacher conducted a "software factory" experiment, giving Astra full autonomy to develop a Python with virtual threads and lexical scoping. After 35 hours and $1200 in API costs (1 billion tokens), Astra produced 75,000 lines of code and 79 commits but yielded nothing of value or any insight into improving the process.
- He observed Astra's bizarre coding habits, such as extensively using Python string manipulation for file edits, even in C and TypeScript contexts, instead of standard tools. This often resulted in highly "codegolfed" and unreadable Python for tool calls.
- This "slop" code, optimized for token efficiency, seeped into committed code, including unit tests, demonstrating a complete disregard for human readability and standard coding styles (e.g., whitespace, indentation).
- Astra also generated highly unorthodox C code, featuring hardcoded constants, random array indexes, and non-standard macro invocations, deviating drastically from established codebase conventions.
- Ronacher posits that this behavior stems from the model's training, which likely rewards long-horizon task completion and token efficiency without adequately penalizing the generation of poor-quality or unreadable code.
- He questions the current direction of AI models, suggesting they might be increasingly optimized for other domains (e.g., lawyers, 3D artists, mathematicians) rather than traditional software engineering, where the astronomical costs don't translate into practical benefits for developers.
- A humorous postscript muses on how models might 'collude' via public wikis in a sandbox environment.
Ronacher concludes that while Astra's raw capabilities are impressive, its impractical and costly output in software engineering leaves him questioning the overall trajectory of AI development and its alignment with real-world developer needs, forcing greater human review for diminishing returns.
The Gossip
Astra's Alarming Automation
Many commenters corroborate the author's experience of Astra producing unreadable, inefficient, or overly complex code, especially when given autonomy. They highlight its propensity to "scope creep," generate excessive tests, or "gold-plate" solutions, leading to high operational costs and extended run times without proportional value. The widespread observation of Astra using Python scripts for simple file modifications instead of standard tools is a recurring point of frustration.
Training Trajectories Troubles
Discussion revolves around the underlying reasons for Astra's peculiar behavior, with several users suggesting a shift in Reinforcement Learning from Human Feedback (RLHF) strategies. The consensus points to a change from optimizing for 'human usefulness' to 'long-horizon task completion,' resulting in models that are technically capable of grand tasks but produce 'weird' or impractical outputs from a human perspective. The author's 'Neijuan' analogy deeply resonates, describing a system that demands more effort without improving output.
Compiler Comparisons Conundrum
A lively debate emerges comparing AI-generated unreadable code to assembly code produced by compilers. Some argue that, like compilers, AI's methods to achieve a result don't need to be human-readable, as long as the end goal is met. Others vehemently reject this analogy, emphasizing that LLMs are non-deterministic, lack stable interfaces, and require human understanding for debugging and maintenance, making a direct comparison to predictable compilers fallacious.
Prompting Perplexities & Prescriptions
Commenters generally agree that effectively using advanced AI models like Astra requires highly specific and meticulously "groomed" prompts, often more detailed than what's needed for human developers. Many share strategies for constraining AI behavior, such as explicitly instructing models not to run full test suites or to prioritize code readability, underscoring the ongoing necessity of significant human oversight and deliberate interaction to manage AI output.
A(G)I's Amorphous Applications
There's significant speculation that these highly advanced AI models might be better suited for domains outside of traditional software engineering, such as 3D art or legal tasks, where the generated 'slop' might be less critical or consumed differently. This leads to skepticism about the "AGI" claims and the practical utility of these models for developers, with many finding the high costs and impractical outputs for coding tasks to be a poor value proposition.