Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
Slotstream ingeniously enables running the massive 104GB Qwen3.8-Flash-Next MoE model on Apple Silicon Macs with limited RAM by streaming model 'experts' from SSD. This clever optimization, built with MLX and Swift, achieves respectable performance (up to 12 tok/s on a 48GB Mac) by intelligently managing memory and offloading. Hacker News found this impressive demonstration of local LLM efficiency and resourcefulness particularly engaging, sparking discussion on similar projects and optimization techniques.
The Lowdown
Carloslfu's slotstream is an impressive open-source project designed to make large language models accessible on consumer-grade Apple Silicon Macs, even those with relatively low RAM. It specifically targets the 125B parameter Qwen3.8-Flash-Next model, which typically requires over 100GB of memory, and allows it to run efficiently by dynamically loading model components from SSD.
- Core Innovation:
slotstreamutilizes 'expert offloading' and 'SSD streaming' to manage the massive 104GB 4-bit weights of the Qwen3.8-Flash-Next model. It keeps a small, dense core of the model resident in RAM while streaming the much larger 'routed experts' and n-gram table from disk. - Performance & Requirements: On a 48GB Mac, it achieves approximately 12 tokens/second with a peak memory usage of around 32GB. While memory constraints are heavily optimized, users need about 110GB of free disk space for the initial download of model weights.
- Ease of Use: The project offers a simple
curl | shinstallation for prebuilt binaries and an Ollama-compatible API for easy integration with existing tools. It also includes anauto-modefor intelligent memory management. - Smart Memory Management: The tool dynamically adjusts its memory footprint, targeting a sweet spot of 33GB, which it identifies as the performance 'knee' where additional RAM yields no further speed benefits. It even shrinks its cache under system pressure and regrows it when resources are available.
- Limitations: The project notes that long prompts can be slow to initialize, and performance figures are primarily validated on a single M5 Pro 48GB machine, with estimates for lower-spec Macs based on extrapolation.
In essence, slotstream represents a significant leap in making large-scale AI models locally runnable on hardware that would otherwise be insufficient, showcasing ingenious memory and resource management strategies.
The Gossip
Duplication Dilemma
Many commenters observed a proliferation of similar projects in the MLX ecosystem for offloading large language models, raising concerns about fragmented effort and the 'Not Invented Here' syndrome. While some called for collaboration or upstreaming to existing projects, the author defended his independent exploration as a drive for performance, committing to maintenance and adding comparative benchmarks.
Readme Refinement
One prominent piece of feedback targeted the project's README, suggesting it was overly verbose and read more like a session log than a concise introduction for new users. The author seemed receptive to this, indicating plans to improve clarity and structure.
Bottlenecks & Beyond
Commenters quickly latched onto the README's candid admission that Hugging Face, rather than user internet speed, is the primary bottleneck for downloading the massive 104GB model weights. This also spurred discussions on potential future hardware innovations, such as 'High Bandwidth Flash,' that could revolutionize memory streaming for such demanding AI workloads.