Show HN: Getting GLM 5.2 running on my slow computer
A single-person project unveils Colibrì, an ingenious C engine designed to run the massive 744B GLM-5.2 Mixture-of-Experts language model on standard consumer hardware, completely without a GPU. It achieves this by cleverly streaming model experts from disk, proving that frontier-class AI can be accessible even on a laptop. This project appeals to HN's love for bare-metal optimization and making powerful tech available to the masses.
The Lowdown
Colibrì is a remarkable open-source project that enables users to run the colossal GLM-5.2 (744B MoE) large language model on consumer-grade computers, even those with limited RAM and no dedicated GPU. Born from the developer's quest to make such a powerful model work on their own modest 25GB RAM laptop, Colibrì demonstrates a highly optimized approach to overcome typical hardware limitations. It's a testament to ingenuity in making cutting-edge AI more widely accessible.
- Core Concept: Colibrì leverages the sparse activation nature of Mixture-of-Experts (MoE) models. Only a small fraction of parameters (~40B) are active per token, and crucially, only ~11 GB of these (the routed experts) change from token to token.
- Memory Management: The fixed 'dense' part of the model (~17B parameters, ~9.9 GB int4) resides in RAM. The vast majority, 21,504 'routed experts' (~370 GB int4), live on disk and are streamed on demand using an LRU cache and OS page cache.
- Technical Stack: The engine is written in pure C (a single 1,300-line file), has zero dependencies, and requires no BLAS, Python runtime, or GPU. It was developed and tested entirely on a 12-core laptop with 25 GB of RAM.
- Key Optimizations: Includes faithful GLM-5.2 forward, MLA attention with compressed KV-cache (57x smaller), DeepSeek-V3-style sigmoid router, native Multi-Token Prediction (MTP) speculative decoding (achieving 2.2-2.8 tokens/forward), true sampling tuned for int4, AVX2 integer-dot kernels for faster matmuls, MLA weight absorption, and async expert readahead.
- Quantization and Conversion: An offline Python converter handles FP8 to int4 conversion shard by shard, ensuring the 756 GB FP8 model never needs to be fully present on disk. The engine also features robust quantization kernels.
- Performance: On the developer's machine, it achieves ~0.05-0.1 tokens/second cold, with estimates reaching 5-15 tokens/second on more powerful CPU-bound machines. A 'learning cache' feature allows the engine to get faster with use by pinning frequently accessed experts.
- Community Involvement: The developer actively seeks benchmark numbers from users with better hardware and assistance with quality benchmarks to measure the accuracy impact of int4 quantization.
Colibrì, named after the hummingbird for its ability to sustain an immense model on minimal resources, truly redefines the boundaries of local LLM inference. It offers a practical path for enthusiasts and researchers to experiment with a frontier-class model without the prohibitive cost of high-end AI accelerators.