Show HN: Understudy – Teach a desktop agent by demonstrating a task once
Understudy is a novel local-first desktop AI agent designed to automate complex, multi-application tasks by simply observing a user perform them once. It moves beyond brittle macros by extracting user intent, allowing it to adapt to UI changes and discover faster execution routes over time. This "Show HN" project, with its initial learning layers already functional on macOS, presents a compelling vision for future desktop automation.
The Lowdown
Understudy introduces an innovative approach to desktop automation, aiming to streamline fragmented workflows that span across various applications, browsers, terminals, and messaging tools. At its core, it's a local-first AI agent runtime that learns by demonstration, transforming a single user interaction into a reusable, intelligent skill.
- Teach-by-Demonstration: The primary feature allows users to perform a task once, while Understudy records screen activity and semantic events. It then extracts the underlying 'intent' rather than just coordinates, creating robust skills that are resilient to UI changes or window resizing.
- Intelligent Task Execution: Unlike simple macros, Understudy can adapt and optimize. Its published skills store intent procedures, various route options (e.g., API calls, CLI commands, browser interactions, GUI clicks), and GUI hints as a last resort, enabling it to prefer faster execution paths when available.
- Five Layers of Autonomy: The system is structured around a five-layer progression mirroring a human's learning journey: (1) Operate Software Natively, (2) Learn from Demonstrations, (3) Crystallized Memory, (4) Route Optimization, and (5) Proactive Autonomy. Layers 1 and 2 are currently fully implemented and usable on macOS.
- Unified Desktop Runtime: Understudy integrates diverse execution routes, including native macOS GUI automation, Playwright-managed browser interactions, shell command execution, web search, persistent memory, messaging across 8 channels, and scheduling, all within a single agent loop.
- Local-First and Model-Agnostic: It operates locally by default, ensuring data privacy, and is designed to be independent of specific AI models, supporting various providers like Anthropic, OpenAI, and Google through a configurable system.
- Current Status and Future: Currently developed for macOS, with cross-platform core features and plans for Linux and Windows GUI backends. Installation is via npm, and the project offers a clear roadmap for advancing towards greater autonomy and proactive task management.
Understudy's ambitious goal is to evolve into a proactive digital colleague that observes, learns, optimizes, and eventually anticipates user needs, fundamentally reshaping how individuals interact with their computer systems by automating complex, multi-application tasks with increasing intelligence and autonomy.