Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

This project showcases an innovative open-source engine, TurboFieldfare, that runs the large Gemma 4 26B language model on Apple M-series Macs with just 2GB of RAM. It achieves this by cleverly streaming model experts from SSD using a custom Swift and Metal runtime. This technical feat pushes the boundaries of on-device AI, making powerful LLMs accessible on commodity hardware, sparking significant interest among developers constrained by memory.

148

Score

Comments

Highest Rank

on Front Page

First Seen

Jul 29, 3:00 PM

Last Seen

Jul 29, 8:00 PM

Rank Over Time

The Lowdown

TurboFieldfare is a specialized inference engine developed by gitpusher42 that enables the Gemma 4 26B-A4B language model to run on Apple M-series Macs using approximately 2 GB of RAM, a remarkable achievement given the model's 14 GB 4-bit quantized weights. The project's core innovation lies in its ability to stream only the necessary routed experts from the SSD, while keeping the shared model components and KV cache in RAM. This approach bypasses the memory limitations typically faced by devices with 8 GB or 16 GB of unified memory.

Key aspects of TurboFieldfare include:

Memory Optimization: It allows a 26-billion-parameter model (Gemma 4 26B-A4B) to run with a memory footprint of about 2 GB by streaming 4-bit quantized weights from SSD.
Custom Runtime: The engine is built with Swift and Metal, specifically tailored for Apple Silicon's shared memory architecture.
SSD Streaming: It leverages bounded parallel pread operations and a small expert cache to efficiently load model parts as needed, with GPU processing concurrent to I/O.
Performance: It achieves 5-6 tokens/second (tok/s) on an 8 GB M2 MacBook Air and 31-35 tok/s on an M5 MacBook Pro.
User Experience: It provides a native Mac app, a command-line interface (CLI), and an experimental OpenAI-compatible local server.
Open-source & Experimental: The project is open-source, detailed with extensive documentation on its optimization journey and experiments.

This project represents a significant step forward in making large, powerful AI models more accessible for local inference on consumer-grade Apple hardware, challenging conventional expectations for on-device AI capabilities.

The Gossip

Performance Peculiarities & Power: Decoding Speed Discrepancies

Users were curious about the significant performance gap between M2 and M5 Macs, expecting SSD performance to be a bottleneck. The author and others clarified that faster SSDs in newer models (M5), potentially more RAM allowing for better OS caching, and even differences between 256GB and 512GB SSD configurations on base models contribute to the observed speed variations. The discussion also touches on how Apple's unified memory architecture influences these results, with free RAM acting as disk cache.

Competing Contraptions & Custom Caching: Comparing with Existing Solutions

A central point of discussion was how TurboFieldfare compares to established local inference solutions like `llama.cpp` and MLX. Commenters inquired about the differences, particularly regarding `mmap` for SSD offloading. The author explained that while `llama.cpp` might run the model in low RAM using `mmap`, TurboFieldfare's specialized approach of explicitly synchronizing `pread` calls with GPU work and its expert caching mechanism offers significant performance advantages over OS-managed `mmap`, leading to much faster token generation. For maximum speed on high-RAM Macs, MLX is still the choice, but TurboFieldfare offers a strong memory-saving trade-off.

Technical Tactics & Thorough Trade-offs: The Engineering Behind Low-RAM LLMs

The community delved into the specific technical innovations enabling this low-RAM inference. Questions arose about the biggest performance gains during development and how frequently experts change, a critical factor for SSD streaming. The author revealed that explicit `pread` over `mmap` was a major win. Despite the full route changing almost every token, an expert cache achieving approximately a 67% hit rate significantly reduces I/O. This detailed insight into the system design, including expert caching and SSD streaming, highlights the clever trade-offs made to optimize performance under severe memory constraints.

Platform Prospects & Practicalities: Hardware Limitations and Future Hopes

Discussions extended to the practical implications and future potential of the project. Concerns were raised about the impact of continuous SSD reads on drive longevity, which the author clarified is not an issue as reads do not cause significant wear. Many expressed hope for cross-platform support, particularly for Windows or iPhone/iPad. The author explained that the heavy reliance on Apple's Metal and shared memory architecture makes it specific to Apple Silicon, though adapting it for iOS/iPadOS is theoretically easier.