Hot Chips 2026: Applying High Bandwidth Flash (HBF)
High Bandwidth Flash (HBF) proposes a novel approach to memory, packaging flash storage like HBM directly onto compute chips to address ever-growing capacity demands, especially for AI workloads. However, HBF fundamentally behaves like a block storage device, not RAM, presenting immense software challenges for adoption. Hacker News is keenly interested in this innovative hardware solution, weighing its potential to alleviate memory bottlenecks against the complex software engineering required to leverage it effectively.
The Lowdown
Hot Chips 2026 introduced High Bandwidth Flash (HBF), a concept leveraging existing flash memory in a High Bandwidth Memory (HBM)-like package directly on a compute chip. This technology aims to offer significantly higher capacity than HBM with decent bandwidth, targeting large machine learning workloads. Still in the simulation phase, HBF's potential hinges on complex software adaptations.
- HBF utilizes flash memory but is physically configured like HBM, sitting on the same package as a compute chip, offering vastly more capacity than HBM. It is distinct from technologies like Intel Optane, functioning more like an integrated SSD.
- Data movement requires Direct Memory Access (DMA) between HBF and DRAM, and accesses must be in large, aligned chunks, akin to a mass storage device, not fine-grained system memory.
- Host software must manage functions typically handled by an SSD controller, such as write leveling and data retention, making HBF far from a plug-and-play solution.
- For machine learning, strategies include storing Mixture-of-Experts (MoE) model experts in HBF, DMA-ing active ones to HBM, or placing the KV cache in HBF for sparse attention mechanisms.
- HBF can also reduce cross-device communication in sharded models by enabling more extensive replication of model weights across GPUs.
- Cost-effectiveness is workload-dependent; HBF is efficient for cost-per-capacity but less so for cost-per-bandwidth compared to HBM, making it unsuitable for bandwidth-bound scenarios.
- The software challenges are substantial, requiring significant changes to existing frameworks due to HBF's block-storage nature, which is comparable to low-level disk access APIs (e.g., O_DIRECT).
The article concludes that while HBF could alleviate DRAM capacity issues, the software hurdles are immense, potentially making streaming model weights from a traditional SSD an easier path due to existing OS abstractions. Its market adoption remains uncertain, pending the successful navigation of these complex technical and cost considerations.
The Gossip
Flash's Fickle Future
Commenters quickly raised concerns about the fundamental properties of flash memory, specifically its durability and write endurance compared to DRAM. While acknowledging that ML workloads are often read-heavy, which might mitigate the impact of lower write endurance, the inherent limitations of flash technology remain a point of discussion regarding the long-term viability and operational characteristics of HBF.
Bandwidth Blips and Bottlenecks
The impressive bandwidth figures cited for HBF, particularly 3TB/s, caused initial surprise and skepticism among users. Discussions highlighted that even with such high bandwidth, the latency and practical implications of accessing large portions of the chip (e.g., 1/6 second to read the whole chip) could limit its effectiveness for streaming model weights, suggesting a potential mismatch between raw bandwidth numbers and real-world performance for certain use cases.