The Memory Wall: Why High-Bandwidth Flash is the Real Test for AI Scaling

In the current AI gold rush, the conversation usually centers on chip architecture and the raw compute power of the latest accelerators. But for those of us tracking actual deployment—the movement of models from research labs into the power-hungry reality of the data center—the real bottleneck isn't the processor. It is the memory.

Until now, high-end AI accelerators have relied almost exclusively on high bandwidth memory (HBM). As The Register first reported, HBM is capable of shuffling data at multiple terabytes per second, but it is severely limited in capacity, typically reaching only into the gigabytes. This limitation forces developers to shard massive models across multiple processors, creating a reliance on high-speed chip-to-chip interconnects that can introduce their own performance bottlenecks.

Enter high-bandwidth flash (HBF). Developed by Sandisk and SK Hynix, HBF is designed to bridge the gap between the massive capacity of SSDs and the blistering speed of HBM. By stacking 16 layers of NAND flash, Sandisk claims its first-generation modules can reach read bandwidths of up to 1.6 TB/s. While this is slower than HBM4—which The Register notes is already hitting 2.5 TB/s per 12-high stack—future iterations of HBF are expected to scale to 2 TB/s and eventually 3.2 TB/s.

From a pragmatic deployment perspective, the bandwidth is impressive, but the capacity is the true disruptor. Sandisk reports that HBF can achieve 256 Gb per die, resulting in 512 GB per 16-high module. To put that in perspective, that is more than 14 times the capacity of the HBM4 currently utilized in the latest accelerators from Nvidia and AMD.

However, we must distinguish between theoretical capacity and operational viability. HBF is not a wholesale replacement for HBM; doing so would be catastrophic. Because HBF utilizes NAND flash rather than DRAM, it suffers from two critical weaknesses: access latency (measured in microseconds rather than the tens of nanoseconds seen in DRAM) and finite write endurance. The Register warns that swapping HBM for HBF would likely lead to rapid hardware failure, potentially turning a GPU costing over $50,000 into a "paperweight"—a risk no hyperscaler is willing to take given their strict depreciation schedules.

Instead, the viable path forward is a hybrid architecture. Sandisk and SK Hynix propose using HBF to supplement HBM, dividing the labor of AI inference into two distinct phases:

1. **The Prefill Phase:** This is the compute-intensive stage involving tokenization, embedding prompts, and generating key-value caches. Because this phase is write-heavy, it remains the domain of HBM. 2. **The Decode Phase:** This is the bandwidth-bound stage where the system reads model weights (or active parameters in mixture-of-experts models) repeatedly to generate tokens. Since this is almost entirely a read operation, HBF is an ideal medium.

In this "write-once, read-many" scenario, the endurance limitations of NAND flash become irrelevant. Furthermore, because HBF is non-volatile, it functions similarly to Intel’s Optane persistent memory; model weights are already present on the device, eliminating the need to reload them from external storage into GPU memory.

The implications for scaling are massive. Sandisk has outlined a potential configuration that would combine six stacks of HBF with two stacks of HBM to reach 3.12 TB of memory. This level of on-chip memory could fundamentally change how models are deployed. Currently, frontier models often use a mixture-of-experts (MoE) architecture to manage memory constraints, spreading "experts" across multiple GPUs. With HBF, even multi-trillion-parameter models—such as Kimi K3—could potentially be housed on a single accelerator, removing the interconnect bottlenecks entirely. On a larger scale, a 72-GPU rack could theoretically support models with hundreds of trillions of parameters.

From a cost and integration standpoint, the outlook is optimistic. HBF modules share packaging requirements with HBM, meaning they can be integrated using established techniques like Intel’s Foveros and EMIB or TSMC’s CoWoS. Crucially, the storage vendor indicates that HBF will not carry a price or power premium over HBM, making it a highly attractive option on a bits-per-dollar basis.

Ultimately, the success of AI deployment won't be measured by how many parameters a model has, but by how efficiently we can serve those parameters without crashing the grid or bankrupting the operator. If HBF can successfully offload the read-heavy decode phase while maintaining HBM-like speeds, it may finally break the memory wall that has constrained the industry.