My local model setup on an M4 Pro Mac Mini
This post details a streamlined setup for running local LLMs on an M4 Pro Mac mini, leveraging tools like oMLX and Tailscale for a powerful, private, and cost-effective AI workflow. The author makes a compelling case for local compute, citing advantages in privacy, cost predictability, and independence from cloud API limitations. Hacker News eagerly dissected the practicalities, debating hardware performance, the economic viability against high-end cloud services, and sharing their own experiences with Apple Silicon and open-source models.
The Lowdown
The author, raybb, shares a detailed guide to his local Large Language Model (LLM) setup on an M4 Pro Mac mini, explaining the technical stack, rationale, and practical applications. He aims to handle 80% of daily AI tasks locally, reserving powerful cloud APIs for only the most demanding needs.
- Hardware and Software Stack: Utilizes an M4 Pro Mac mini (48GB RAM) with Qwen3.6-35B-A3B-OptiQ-4bit (reasoning) and Gemma-4-E4B-it-OptiQ-4bit (light chats), oMLX for inference, and Tailscale for secure device networking.
- Motivation for Local: Primary drivers include cost predictability (avoiding expensive cloud subscriptions), data privacy, AI sovereignty (control over model access), improved latency, offline capability, and freedom from API rate limits.
- Usage Scenarios: The Mac mini serves as a central hub for an agent backend (Hermes), quick chats via Apollo on iOS, coding assistance with Pi, and general queries through Raycast AI, accessible across devices via Tailscale.
- Model Nuances & Hardware Compatibility: Explains the critical difference between total parameters and active parameters in Mixture-of-Experts (MoE) models, like Qwen's A3B, which significantly impacts RAM requirements and practical usability on consumer hardware. Provides guidelines for checking model fit.
- Ease of Model Swapping: Highlights the convenience of oMLX's auto-discovery and integrated HuggingFace browser, allowing for simple download and activation of new models, keeping the setup agile with rapid model advancements.
- Networking & Performance: Tailscale creates a secure private network, enabling seamless access from various devices. oMLX's KV cache persistence optimizes performance for agent workflows by restoring context quickly from disk.
The article concludes that local models on Apple Silicon are now a robust and practical solution for many AI tasks, offering significant control and efficiency, with the author already planning an upgrade to an M5 Max Mac Studio due to positive experiences.
The Gossip
Performance Predicaments and Silicon Speeds
Many commenters immediately questioned the actual performance (tokens/second) of local LLMs on Apple Silicon, contrasting their own disappointing experiences with the author's positive assessment. While some reported decent speeds with specific quantized models on M4 Pro (e.g., 52-72 tok/s), others highlighted significant prefill latency or found their M1/M2 setups barely adequate. The discussion also veered into the comparative power of dedicated Nvidia GPUs and high-end hardware like DGX Sparks, suggesting that Apple's unified memory, while good, still faces limitations for truly demanding, interactive agent workloads compared to specialized AI accelerators.
Cloud vs. Local: Cost-Benefit Conundrum
A significant debate emerged around the economic viability and practical superiority of running local LLMs versus relying on cloud APIs. Enthusiasts of local setups championed data privacy, AI sovereignty, and long-term cost savings for high-volume usage, arguing that cloud APIs are "rented land" with unpredictable pricing and hidden data uses. Conversely, skeptics maintained that for most interactive tasks, current cloud models (like Claude Code) offer superior quality and performance that outweighs the cost, especially considering the substantial capital expenditure and effort required for a high-performing local rig. One commenter vigorously argued that for truly maximal, 24/7 AI usage, a local cluster *is* more cost-effective than maxing out API limits, despite the initial investment.
Model Musings and Benchmark Battles
Commenters delved into specifics about various LLMs and their real-world capabilities. There was interest in newer models like Qwen 3.8 27B and DeepSeek (DS V4 Flash), with discussions touching upon how their performance on benchmarks (e.g., DeepSWE) translates to practical applications, particularly for coding tasks. Some noted discrepancies between benchmark scores and actual user experience, suggesting that consistency, rather than peak performance, is crucial for complex projects.