Writing

Memory is the bottleneck nobody budgets for

Memory is the one resource that often gets overlooked in infrastructure planning. Compute and networking are carefully measured, but memory sits in a rigid per-node allocation that creates waste. Every server has its own DRAM, and when that memory isn't fully used, you either strand it or overprovision the next node to compensate. Accelerators like GPUs make this worse: their high-bandwidth memory is scarce and expensive. Once it's filled with large data structures, you're forced to buy more GPUs, not more memory. That leads to hopelessly oversubscribed accelerators where compute is idle waiting on data, but you can't add capacity without swapping out entire devices.

CXL changes the model entirely. Compute Express Link is a cache-coherent interconnect that lets you treat memory as a shared, composable pool rather than a fixed silo on each node. With CXL, a rack full of memory can be dynamically carved out and attached to any host or accelerator over a high-speed fabric. This means you no longer have to match memory to compute at purchase time; you can add capacity independently and reallocate it as workloads change. The result is far less stranded DRAM and much better utilization of expensive accelerators.

This pooled architecture is especially compelling for retrieval-augmented generation and vector search pipelines. When you run a large language model, its key-value cache for attention operations can easily balloon to hundreds of gigabytes, eating into the precious HBM that the GPU needs for active computation. The same goes for dense vector embeddings in a vector database: they consume space that could be used for batching and interleaving compute. By offloading these data structures to CXL-attached memory, you free up GPU HBM for the matrix multiplies that actually drive inference, while keeping the KV caches and embeddings in a fast, coherent address space.

The natural question is whether adding a hop over CXL introduces unacceptable latency. In practice, the access patterns of KV caches and bulk vector reads are largely sequential and bandwidth-sensitive, not latency-critical. A well-timed prefetch can hide much of the round-trip cost. CXL.mem latencies of a few hundred nanoseconds are far lower than going to NVMe storage or even remote NUMA nodes, yet they give you terabytes of capacity that HBM cannot match at any reasonable price. You trade a small increase in access time for a massive increase in dataset size, which directly improves recall in vector search and reduces thrashing in attention caches.

When you adopt this mindset, memory stops being a per-machine limit and becomes a utility you can dial up or down. You can pack more models onto a single GPU, or run larger batch sizes without hitting out-of-memory errors. You can also consolidate vector indexes that would otherwise sit on separate, underutilized servers. The result is a hardware configuration that bends to the needs of the workload, not the other way around. CXL 3.0's multi-host capabilities and fabric-attached memory pools make this vision practical at rack scale, and the software ecosystem is quickly catching up.

I recently unpacked the full architectural details of CXL memory pooling in a piece on The Diligence Stack, for those who want to go deeper into the coherent fabric and latency numbers. But the core takeaway is simple: memory is the bottleneck nobody budgets for, and CXL is the first real chance to fix it cleanly.

Originally covered at thediligencestack.com ↗