Typesafe-computer-use drives a Mac toward a goal for 1/50th of a cent per step
This GitHub project introduces typesafe-computer-use, a macOS automation tool designed for extreme cost-efficiency and speed. It orchestrates computer actions using OCR, accessibility data, and a specialized small decision model, significantly outperforming large language models in cost and latency per step. By selectively engaging powerful (and costly) LLMs only when free-text generation is genuinely required, it offers a pragmatic blueprint for building affordable, high-performance AI agents.
The Lowdown
The typesafe-computer-use project presents an innovative approach to computer automation on macOS, aiming to overcome the high costs and latency associated with using large language models (LLMs) for every step. Instead of continuously sending screenshots to expensive frontier models, this system intelligently combines deterministic screen reading, accessibility tree analysis, and a small, fast decision classifier to achieve user-defined goals at a fraction of the cost.
Here's a breakdown of its core features and methodology:
- Cost and Performance Advantage: The system boasts an operational cost of approximately $0.0002 per step, making it 155x to 390x cheaper and 14x to 40x faster than relying on LLMs like Claude Opus for each decision. End-to-end steps are also significantly quicker, around 1.5 seconds compared to 5.5 seconds.
- Hybrid AI Approach: It leverages macOS's native Vision OCR for screen content and its accessibility tree for interactive elements. A specialized decision model called TypeSafe rapidly selects the next action from a predefined, mutually exclusive set based on this input. A larger 'writer model' is only invoked for generating free text (e.g., filling forms) or providing final answers, ensuring cost-intensive resources are used judiciously.
- Action Space: The system supports various actions including clicking items, using browsers (with URL management), typing text, pressing specific keys (enter, escape), scrolling, and waiting, all determined by the decision model.
- Intelligent Screen Perception: OCR processing is optimized by cropping to active window regions and reusing unchanged screen tiles, drastically reducing the amount of text processed. The accessibility tree is pruned and filtered to ensure relevant and actionable elements are identified, even for off-screen controls.
- Deterministic and Debuggable: Every run generates a detailed folder containing logs, raw and annotated screenshots, and decision payloads, allowing for thorough offline replay and debugging. This transparency helps identify and fix automation stalls.
- Platform Agnostic Design: The architecture separates platform-specific interactions (e.g., macOS's Quartz, AX, AppleScript) into a dedicated module, hinting at potential future portability to other operating systems like Linux by replacing this component.
By strategically compartmentalizing tasks and only employing powerful, expensive LLMs for their unique strengths, typesafe-computer-use offers a highly efficient, cost-effective, and transparent framework for automating complex computer interactions, proving that smart design can significantly reduce the overhead of AI-driven agents.