Near-Memory Computing Is Back: Why AI Infrastructure Is Moving Closer to the Data

Introduction

Modern AI infrastructure is usually described in terms of GPUs, accelerators, and raw compute capacity. That framing misses a growing constraint: many production workloads are not limited by arithmetic throughput alone. They are limited by how quickly systems can move data into the right place, keep large working sets in memory, and avoid wasting power shuttling bytes across buses, sockets, and networks.

Near-memory computing addresses that problem by moving selected processing closer to memory. Instead of treating DRAM as a passive component attached to a CPU or GPU, new architectures use memory expansion, CXL fabrics, and lightweight compute engines near the data path. The goal is not to replace CPUs or GPUs. The goal is to reduce the amount of unnecessary data movement that makes large-scale AI, analytics, and HPC workloads expensive to run.

This idea is not new. Processing-in-memory and near-data computing have been discussed for decades. What is different now is the combination of AI inference pressure, vector search, KV-cache growth, larger scientific datasets, and the emergence of Compute Express Link as a practical interconnect for memory expansion and pooling.

The Problem: Data Movement Is Expensive

A processor can only work on data it can access. In small systems, that sounds simple. In large AI and HPC systems, it becomes one of the hardest engineering problems.

Every time data moves from storage to memory, from memory to CPU, from CPU to GPU, or across a network fabric, the system pays a cost in latency, bandwidth, power, and complexity. GPUs may deliver enormous math throughput, but they still depend on memory hierarchy, interconnect bandwidth, and orchestration from the rest of the platform.

This is especially visible in workloads such as retrieval-augmented generation, vector search, graph analytics, large-context inference, database acceleration, and scientific simulation. These workloads often touch large datasets where the bottleneck is not always the number of available compute cores. It is the cost of feeding those cores efficiently.

What Near-Memory Computing Means

Near-memory computing places compute capability close to memory resources. That can mean several different architectures:

simple processing engines attached to DRAM controllers
CXL devices that combine memory expansion with local compute
memory pooling systems that expose large shared memory capacity across multiple hosts
accelerators designed for reductions, filtering, vector operations, or preprocessing
software runtimes that schedule work where the data already resides

The common theme is locality. If a workload needs to scan, filter, score, rank, reduce, or preprocess a large dataset, it may be more efficient to perform part of that work close to memory and return a smaller result to the host processor.

This does not make near-memory devices general-purpose replacements for CPUs. Instead, they act as infrastructure components for memory-bound work. The CPU remains responsible for control flow, scheduling, operating system behavior, and general application logic. GPUs remain essential for dense matrix math and model execution. Near-memory compute fills the gap where moving data is the expensive part.

Why CXL Changes the Discussion

Compute Express Link gives system architects a standardized way to attach memory and accelerators over PCIe while preserving coherent memory semantics. That matters because previous memory-expansion approaches were often proprietary, application-specific, or difficult to integrate into mainstream server designs.

With CXL, a server can access external memory devices more naturally. At rack scale, CXL fabrics can support pooled memory designs where capacity is not stranded inside individual servers. Instead of overprovisioning every node for peak memory demand, infrastructure teams can allocate memory more flexibly across systems.

That is important for AI because memory demand is uneven. One workload may need large KV-cache capacity. Another may need fast access to vector indexes. A scientific workload may need a large graph or molecular dataset resident in memory. Static server-local memory forces teams to buy for worst-case scenarios. Shared and expanded memory can improve utilization if latency and software integration are handled correctly.

Why AI Inference Makes This More Urgent

Training gets attention, but inference creates persistent infrastructure pressure. A production inference system must serve requests continuously, often with strict latency targets. As context windows grow and retrieval systems become more common, memory usage becomes a central scaling factor.

KV cache is a good example. Large language models store intermediate attention data so they do not recompute everything for every generated token. Longer contexts and more concurrent users increase the memory footprint. If the KV cache cannot be kept close enough to the compute path, latency rises and GPU utilization can fall.

Near-memory systems can help by keeping larger working sets accessible and by offloading selected operations such as retrieval, scoring, filtering, or preprocessing. In some designs, the near-memory device does not need to run the entire model. It only needs to reduce the amount of data that must travel back to the CPU or GPU.

The Enterprise Infrastructure Angle

For enterprise teams, the practical question is not whether near-memory computing is interesting. It is whether the architecture reduces cost, improves utilization, or unlocks workloads that are difficult to run on conventional servers.

Potential advantages include:

larger effective memory capacity for memory-bound AI and analytics workloads
reduced data movement between CPU, GPU, storage, and network layers
better utilization of expensive accelerator infrastructure
lower power use for selected memory-intensive operations
more flexible rack-scale resource allocation
improved support for datasets that exceed a single server’s local DRAM

The tradeoffs are equally important. CXL memory is not identical to local DRAM in latency. Software stacks must understand placement, scheduling, and failure behavior. Observability tooling must expose where memory lives and how it is being used. Procurement teams must also evaluate whether a CXL architecture simplifies operations or introduces another specialized infrastructure layer.

Where It Fits in the Stack

Near-memory computing is most relevant when the workload has a high ratio of data movement to useful computation. That includes vector databases, recommendation pipelines, graph analytics, large-scale search, some database operations, and parts of AI inference.

It is less useful when workloads are dominated by dense compute that already maps efficiently to GPUs or specialized accelerators. In those cases, the main goal remains feeding the accelerator efficiently, not replacing it.

The likely future is hybrid. CPUs coordinate. GPUs perform dense model execution. CXL-attached memory expands capacity. Near-memory compute performs targeted operations close to large datasets. Storage tiers provide persistence. Orchestration systems decide where workloads should run based on latency, capacity, and cost.

Risks and Limitations

Near-memory computing still has several hurdles:

programming models must be simple enough for real adoption
performance claims need independent validation across real workloads
CXL fabrics add topology and management complexity
memory pooling must avoid noisy-neighbor problems
security boundaries become more important when memory is shared
operators need mature monitoring and debugging tools

These issues do not make the architecture unrealistic. They simply mean adoption will likely begin in specialized environments: hyperscale AI infrastructure, research labs, HPC centers, financial analytics, and large enterprise data platforms.

Why This Matters

The industry is moving beyond a simple “more GPUs” answer. AI infrastructure now depends on memory capacity, bandwidth, data locality, networking, cooling, software orchestration, and cost per useful token or query. Near-memory computing is part of that larger shift.

The most important change is conceptual. Memory is becoming an active infrastructure layer, not just a component inside a server. As CXL matures, data centers may treat memory capacity more like a pool that can be expanded, allocated, and optimized across workloads.

For organizations building serious AI systems, this matters because bottlenecks often appear where the architecture is least flexible. If data movement becomes the limiting factor, then solving it requires more than faster processors. It requires redesigning where memory sits, how it is shared, and which operations should happen near the data.

Key Takeaways

AI infrastructure is increasingly constrained by memory movement, capacity, and bandwidth.
Near-memory computing moves selected processing closer to large datasets instead of routing all work through CPUs or GPUs.
CXL makes memory expansion and pooling more practical by giving servers a standardized coherent memory interface.
The strongest use cases include vector search, KV-cache handling, graph analytics, database acceleration, and scientific workloads.
Near-memory compute complements CPUs and GPUs rather than replacing them.
Adoption will depend on software maturity, observability, security, and proven workload-specific performance.
The broader trend is clear: memory is becoming a programmable infrastructure layer.