Samsung's Processing-in-Memory (PIM)
Samsung is pushing the envelope with Processing-in-Memory (PIM) for LPDDR5X, embedding compute units directly into memory chips to dramatically increase bandwidth for tasks like AI/ML. While this offers impressive theoretical performance gains by circumventing traditional memory bottlenecks, it introduces profound challenges for existing hardware and software paradigms. The deep dive reveals how PIM repurposes standard DRAM commands but highlights the extensive systemic changes required for practical, performant adoption.
The Lowdown
Samsung's Processing-in-Memory (PIM) initiative, as discussed at Hot Chips 2026, aims to overcome the memory wall by integrating computation directly into memory chips. This approach leverages the high internal bandwidth of DRAM, allowing for significantly faster data processing for specific workloads, primarily targeting AI/ML applications.
- PIM involves embedding Multiply-Accumulate (MAC) units within LPDDR5X chips, specifically at each DRAM bank, to utilize internal bandwidth up to 614 GB/s.
- The LPDDR5X-PIM design maintains compatibility with standard memory controllers by using special row addresses to switch between normal memory access and PIM compute modes.
- In PIM mode, specific registers for instructions, activation vectors, and scale factors are written across all banks, allowing for SIMD-like operations using low-precision formats (INT8, FP8).
- A single LPDDR5X-PIM chip can deliver 2.4 TOPS, which can be scaled by using multiple chips, though this increases system memory cost.
- Significant software and system integration hurdles exist, including the need for isolated memory regions, impacting multitasking and system responsiveness.
- PIM breaks CPU caching, prefetching, and speculative execution models, requiring memory to be uncacheable and non-speculative, which severely degrades CPU performance.
- The article proposes future hardware changes like expanded DRAM interfaces, memory controllers acting as cache-coherent peer CPUs, and new CPU instructions to enable seamless PIM adoption and avoid current performance compromises.
Ultimately, Samsung's PIM demonstrates a creative way to achieve immense computational throughput within memory. However, its current implementation fundamentally disrupts established computing paradigms, suggesting that widespread adoption will require not just new hardware from memory manufacturers, but also significant architectural shifts in CPUs, memory controllers, and operating systems.