HN
Today

Breaking the 1.58-bit Barrier for Ternary LLMs

This paper dives deep into optimizing the storage and performance of ternary Large Language Models, which use weights of -1, 0, or +1. It introduces a novel packing scheme, BITCOS, that shatters the conventional 1.58-bit barrier by exploiting the actual distribution of weights, especially the abundance of zeros. This is popular on HN because it offers tangible, significant efficiency gains for deploying quantized LLMs, a hot topic for reducing inference costs.

17
Score
0
Comments
#2
Highest Rank
14h
on Front Page
First Seen
Sep 16, 9:00 PM
Last Seen
Sep 17, 10:00 AM
Rank Over Time
522334587911131516

The Lowdown

A new research paper presents a clever method to squeeze more performance out of ternary Large Language Models (LLMs) by rethinking how their weights are stored. Moving beyond the theoretical 1.58-bit per weight barrier, this work addresses a practical inefficiency in existing ternary LLM implementations.

  • The Problem: Ternary LLMs use weights of {-1, 0, +1}, conventionally assumed to be equiprobable, leading to a theoretical storage cost of log2(3) \u2248 1.585 bits per weight. Current implementations often pack five weights into one byte, resulting in an effective 1.625 bits per weight, assuming this equiprobable distribution.
  • The Observation: The authors analyzed 29 ternary LLM models and discovered that zero weights are far more common than assumed, accounting for up to 51.5% of all weights. This non-uniform distribution means the current packing methods are suboptimal.
  • The Solution: BITCOS: The paper introduces "BITCOS," a simple yet effective distribution-adaptive storage layout. It uses a dense presence bitmap (indicating whether a weight is zero or non-zero) and a compacted sign vector (for the non-zero weights). This design allows for storage at 2-z bits per weight, where z is the zero density.
  • Performance Gains: BITCOS achieved greater compactness than the standard five-trit packing in 26 out of 29 tested models, reaching as low as 1.485 bits per weight. It is designed for efficient unpacking on modern hardware, with optimized sequences for AVX-512, AVX2, and Intel Xe2 GPUs. Measurements showed up to a 1.28x gain in ternary matrix-vector multiplication kernels and up to 1.18x to 1.27x improvement in end-to-end LLM decode throughput across various CPU and GPU platforms.

By intelligently exploiting the prevalence of zero weights in real-world ternary LLMs, BITCOS offers a practical and significant improvement in both storage efficiency and inference performance, pushing the boundaries of what's possible for resource-constrained LLM deployments.