Show HN: TUI-use: Let AI agents control interactive terminal programs

Introducing tui-use, a tool enabling AI agents to control interactive terminal programs like REPLs, installers, and TUI apps, bridging a significant gap in agent capabilities. It allows agents to observe and interact with terminal screen output as clean text, much like a human would. While some question its novelty against existing terminal multiplexers, others hail it as a crucial step for expanding AI automation into human-centric command-line interfaces.

Score

Comments

#12

Highest Rank

on Front Page

First Seen

Apr 8, 5:00 PM

Last Seen

Apr 8, 7:00 PM

Rank Over Time

The Lowdown

TUI-use is an innovative tool designed to extend the capabilities of AI agents by allowing them to interact with and control text-based user interface (TUI) applications, REPLs, and interactive command-line programs. Traditionally, AI agents excel at running shell commands or parsing standard output, but they falter when faced with programs requiring interactive input or navigating complex TUI screens.

Core Functionality: TUI-use operates by spawning programs in a pseudo-terminal (PTY), rendering their output with a headless xterm emulator, and providing agents with clean text 'snapshots' of the terminal screen. This includes metadata like highlights for selected items and is_fullscreen status, allowing agents to understand the visual state of the TUI.
Enhanced Interaction: It enables agents to send rich key presses (e.g., Enter, Ctrl+C, arrow keys) and text, effectively mimicking human interaction with terminal applications.
Broad Use Cases: This opens up possibilities for agents to handle interactive REPL sessions (Python, Node), step through CLI scaffolding tools (npm create), manage database CLIs (psql, mysql), operate programs over SSH, and navigate full-screen TUI apps like vim or htop.
Integration Focus: The project emphasizes integration with existing AI coding agents, offering a Claude Code plugin as a first example, with plans for more.
Technical Details: It uses a daemon to manage PTY sessions and a snapshot model, ensuring agents receive a clear, reassembled view of the terminal state rather than raw byte streams.
Limitations: Currently, it's Unix-only (macOS/Linux) and sacrifices some color/style information, prioritizing clean text and structural indicators.

TUI-use effectively transforms complex terminal visuals into actionable, structured data for AI, promising a new era of agent autonomy within a human-designed terminal landscape.

The Gossip

Terminal Tool Talk: Tmux vs. tui-use

A significant portion of the discussion revolved around whether `tui-use` reinvents existing solutions, particularly `tmux`. Commenters questioned what new capabilities `tui-use` offers beyond what tools like `tmux` already provide for programmatic terminal control. Some expressed a view that this approach might be an 'AI-era' rediscovery of long-standing Unix philosophies, potentially adding unnecessary complexity instead of leveraging established piping mechanisms for agent interaction.

Agent Aspiration: Expanding AI's Terminal Reach

Despite the skepticism, other users recognized `tui-use` as an exciting development for AI agents. They highlighted its potential to overcome current limitations where agents struggle with interactive prompts and TUI applications, which are ubiquitous in development workflows. This project is seen as a stepping stone towards more general and capable AI agents that can seamlessly operate in environments traditionally reserved for human interaction.

Practical Plugin Possibilities

Commenters quickly jumped to envisioning practical applications for `tui-use`. One notable suggestion was to use it to interact with interactive AI sessions (e.g., Claude) to bypass API costs, effectively using a 'max subscription' interactively rather than paying for API credits. This illustrates the desire for tools that can democratize AI agent access and utilization by enabling interaction with consumer-facing interfaces.