Samsung Targets AI Inference Costs With PIM Memory Integration

Samsung is attempting to reshape the cost-per-token equation for AI inference by integrating compute capabilities directly into memory chips. As reported by Chips and Cheese, Samsung presented its Processing-in-Memory (PIM) push at Hot Chips 2026, showcasing an LPDDR5X-PIM architecture designed to eliminate the high-latency path and bandwidth bottlenecks found between traditional compute cores and DRAM.

While standard DRAM accesses are limited by the external interface—maxing out at 76.8 GB/s—Samsung’s LPDDR5X-PIM places a PIM block at each of the chip's 16 banks. By accessing these banks without external bus constraints, the system can utilize an internal bandwidth of 614 GB/s. Each PIM block features a MAC tree supporting low-precision formats, including INT8 and FP8. According to Samsung, a single chip can reach 2.4 TOPS of package-wide compute throughput; a configuration of eight 16 GB LPDDR5X chips would provide 9.6 INT8 TOPS, roughly matching the NPU in Intel’s Meteor Lake.

To maintain compatibility with existing systems, Samsung designed the hardware to stay within the standard LPDDR5X protocol. The company achieves this by using special row addresses that function as MMIO addresses, allowing the chip to switch between a standard single-bank mode and a multi-bank mode. In multi-bank mode, the chip operates as a constrained SIMD processor, broadcasting operations, scale factors, and source operands across all 16 banks simultaneously.

For fintech LLMs and other AI workloads, this shift moves the heavy lifting of matrix multiplication into the memory itself. By reducing the need to constantly shuttle massive model weights across the system bus, Samsung's architecture targets the primary energy and performance overheads associated with large-scale AI inference.