Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

RunAnywhere, a YC W26 startup, has unveiled RCLI, an on-device voice AI pipeline for macOS designed for ultra-low latency and enhanced privacy. This system integrates Speech-to-Text (STT), Large Language Models (LLM), and Text-to-Speech (TTS) to deliver sub-200ms end-to-end responses, facilitating local AI interactions like voice control for macOS actions and Retrieval-Augmented Generation (RAG) over personal documents, all without cloud dependencies.

• Core Technology: The system leverages MetalRT, a proprietary GPU inference engine optimized for Apple Silicon. It bypasses conventional inference engine overheads by employing custom Metal compute shaders and pre-allocating memory to eliminate runtime allocations during inference.
• Performance Benchmarks: RCLI claims impressive speedups, reporting LLM decoding 1.67x faster than llama.cpp and 1.19x faster than Apple MLX, alongside STT that is 4.6x faster than mlx-whisper.
• Key Features: It offers a complete voice pipeline (VAD, STT, LLM, TTS), supports 43 macOS actions controllable by voice, provides local RAG for document querying, and includes an interactive Terminal User Interface (TUI) for system management and benchmarking.
• Open-source & Proprietary Blend: While RCLI itself is open-source (MIT license), the underlying MetalRT engine is proprietary and requires M3+ chips for optimal performance, gracefully falling back to llama.cpp on M1/M2 devices.
• Addressing Latency: The project's primary goal is to resolve the challenge of compounding latency in sequential voice AI pipelines, aiming to make on-device AI genuinely fast and practical.

In essence, RunAnywhere presents a significant advancement towards fully local, high-performance voice AI on Apple hardware, positioning itself as a robust solution for low-latency, privacy-focused applications that avoid reliance on cloud services.