AI hits memory wall, now needs new context tier

Presented by Solidigm As inference workloads evolve from discrete question-and-answer exchanges into persistent, multi-step agentic systems, GPU availability is no longer the most critical AI bottleneck. Instead, the bottleneck has migrated from compute to context, says Jeff Harthorn, AI applied research lead at Solidigm. "Why context management has become a primary bottleneck, more than GPU availability or compute efficiency, is the question of 2026," says Harthorn.

"GPUs have gotten dramatically cheaper per FLOP. Model architectures and inference serving engines have all gotten much more efficient. But the thing that's grown faster than both of those is context.

The persistent state that has to live between sessions has grown even faster than context itself." It's happening as context windows grow dramatically, making individual inputs far larger than before. Agentic AI systems chain dozens or hundreds of model calls together, each generating state that must be tracked, and enterprises are requiring that inference state persist across sessions for audit, governance, and reuse. These trends compound each other, pushing context volumes beyond what any existing memory tier was designed to handle.

"Those three things are all happening at the same time, all of which are pushing context data and context memory into the stratosphere much more quickly than we're used to seeing," adds Ace Stryker, director of AI and ecosystem marketing at Solidigm. The solution is a dedicated context tier emerging between GPU memory and bulk network storage: a layer of high-performance, high-density flash designed specifically to hold and serve Key-value (KV) cache, the inference data that allows models to retain and reuse context, and retrieval data at inference speed. Nvidia has formalized this architecture under the term CMX.

Storage companies including Solidigm are building SSD products optimized for this workload. "Storage has not been the first thing folks have thought about when they've been planning their enterprise infrastructure buildout," Stryker says. "In a lot of ways, it was a relatively small cost compared to compute, and it was a commodity.

You just shopped around for the lowest dollar per gigabyte and called it good. But now, if your storage is not up to snuff, your ROI suffers, and it directly impacts your bottom line.” Why AI inference requires a different storage architecture than training The storage architecture that AI systems rely on today was largely inherited from training workflows. Training is sequential and write-dominated, with data moving in large blocks to and from bulk object storage.

The tier structure, with high-bandwidth memory on the GPU, fast NVMe in the server, and bulk storage over the network, serves that use case reasonably well. However, inference is a different animal. Its I/O signature is fine-grained, latency-sensitive, and increasingly stateful.