Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
WASTE, a C inference engine, cleverly enables the massive 2.78-trillion-parameter Kimi K3 LLM to run on consumer hardware by streaming model 'experts' directly from NVMe. This ingenious approach, despite a low 0.5 tok/s throughput, fundamentally shifts the discussion from the feasibility of local frontier models to engineering challenges. The technical feat sparks deep dives into performance trade-offs, power consumption, and even spirited debates about LLM-assisted documentation writing.
The Lowdown
WASTE is an innovative C inference engine designed to push the boundaries of local large language model (LLM) deployment, specifically enabling the colossal 2.78-trillion-parameter Kimi K3 model to run on consumer-grade hardware. It achieves this by intelligently streaming model weights from NVMe storage, effectively making frontier-scale AI accessible outside of data centers.
- The engine allows the 2.78-trillion-parameter Kimi K3 to run on as little as 29 GB of RAM (minimum, 64 GB recommended for practical use) and nearly 1 TB of NVMe storage.
- It operates at approximately 0.50 tokens per second (tok/s) by keeping only the core model in memory and streaming "experts" (portions of the Mixture of Experts architecture) from disk as needed.
- Key technical innovations include optimizing disk I/O (one
preadper expert), utilizing 3-bit residual vector quantization for experts, and a sophisticated cache management strategy to prevent OS-level paging. - The project prioritizes demonstrating the feasibility of running such large models locally over achieving high speed, asserting that the remaining challenges are engineering rather than fundamental.
- WASTE is self-contained, dependency-free (at runtime), fully embeddable, and includes an OpenAI-compatible HTTP server, as well as multimodal support for image processing.
- The project also provides tools for converting the Kimi K3 model from its original format, detailing the specific hardware requirements and performance considerations for optimal operation.
By proving that models like Kimi K3 can be "reachable" on single consumer machines, WASTE opens new avenues for private, offline AI applications, transforming the conversation around large model deployment from an impossible dream for many to a tangible, albeit slow, reality.
The Gossip
AI's Authorship Anxieties
This theme discusses the style and origin of the README, with many users speculating and some criticizing that it was largely generated by an LLM, leading to a discussion about the role of AI in writing documentation and even committing code. While some found it hard to understand or
Performance Ponderings & Practicality Paradox
Commenters extensively debate the practical utility and energy efficiency of an LLM running at 0.5 tokens/second, questioning whether such a slow speed offers meaningful real-world application, especially when contrasted with cloud service efficiency. Discussions delve into potential use cases, the energy cost per token, and overall viability.
Technical Commonalities & Comparisons
Users ask about the project's relationship to existing or similar initiatives, seek clarification on specific technical choices, and inquire about the availability of pre-converted models. This includes comparisons to other local LLM efforts and questions about hardware-specific optimizations like Apple's Metal.