I'm a seeing-eye dog for a computer
Clayton Ramsey, a robot programmer, details his frustrating attempts to leverage large language models for visual debugging of robot code. Despite their coding prowess, these AI assistants demonstrate a profound inability to interpret physical robot behavior or effectively interact with debugging GUIs. This candid account resonates with developers facing the practical limitations of current AI tools, offering a reality check on LLM capabilities beyond text generation.
The Lowdown
The author, Clayton Ramsey, recounts his experience attempting to integrate large language models (LLMs) into his workflow for debugging robot code. He envisioned a future where LLMs would handle the tedious visual inspection and correction cycles, but reality proved far more challenging.
- Ramsey, a robot programmer, uses LLMs to assist with code generation but identifies visual debugging as a particularly "drudgy" and inefficient process.
- He aimed to automate this visual debugging loop, where he would normally manually adjust "magic numbers" based on robot visualization output, by having an LLM interpret the visual feedback.
- His attempts revealed significant shortcomings: LLMs lack an inherent understanding of how a "normal, working robot does" things and struggle with controlling GUI debugging tools effectively.
- The LLMs' inability to navigate the visualizer efficiently meant simple tasks, like getting to the correct view, took minutes instead of seconds, leading to thirty-minute waits for consistently wrong answers.
- Ultimately, Ramsey found himself in a "dumbest workflow possible" where he became the "seeing-eye dog" for the computer, manually interpreting screenshots of failed robot actions and relaying detailed instructions back to the LLM.
- He concluded that he might as well do the debugging himself, as it was less frustrating and he'd at least get to enjoy the "fun part" of the process.
The story serves as a cautionary tale regarding the current limitations of LLMs in highly visual, physical, and interactive domains like robot debugging, highlighting the gap between their textual capabilities and practical application in complex real-world tasks.