Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

Inspired by a previous Hacker News post about running Gemma 4 on a 10-year-old Xeon, the author embarked on a similar challenge: making Google's Gemma 4 26B mixture-of-experts model run on a 13-year-old repurposed HP StoreVirtual server. This server, equipped with dual Ivy Bridge Xeons and no GPU, lacked modern instruction sets like AVX2 and FMA3, leading to unexpected compilation and runtime issues.

• The Initial Hurdle: The ik_llama.cpp fork, optimized for newer CPUs, failed to build due to assumed AVX2/FMA3 instructions not present on the older Ivy Bridge Xeons.
• AI as a Collaborator: Instead of traditional debugging, the author fed the build failures and subsequent experimental results to Claude, an AI agent. Claude successfully diagnosed the issue: specific MoE graph operations (MOE_FUSED_UP_GATE and FUSED_UP_GATE) had no dispatch case for non-AVX2 builds, causing them to silently use uninitialized memory.
• The 'Silent Bug': This led to "fluent-looking multilingual gibberish" output, characterized by uniformly spread token IDs and a flat softmax due to a hidden state being consistently corrupted. The determinism of this corruption (temperature 0, byte-identical runs) helped Claude identify the bias.
• The Fix: Claude generated three commits: compile fixes for scalar fallback paths, a runtime fix redirecting unsupported ops to working non-IQK implementations via ggml_mul_mat_id and ggml_fused_mul_unary, and updates to CI stubs.
• Performance: With these fixes, the 26B MoE model achieves approximately 5.2 tokens/second for decoding and 16 tokens/second for prompt evaluation.
• Broader Implications: The author emphasizes that the real skill lies not in subscribing to AI services, but in understanding models deeply enough to adapt them to challenging, non-standard environments, and leveraging AI as a powerful diagnostic and problem-solving tool rather than a mere 'fix-it' button.

This story powerfully illustrates that powerful AI models can be made accessible on surprisingly modest hardware with enough ingenuity and, increasingly, with the help of other AIs. It serves as a compelling example of pushing the boundaries of what's possible with legacy systems and highlights the evolving human-AI collaborative workflow in complex engineering tasks.