Low-overhead General-purpose Near-Data Processing in CXL Memory Expanders

TL;DR

This work tackles the bottlenecks of memory-bound workloads on systems using CXL memory by introducing memory-mapped near-data processing, $M^2$NDP. It fuses a low-overhead host-communication mechanism, $M^2$func, with a lightweight, highly concurrent execution model, $M^2$μthread, implemented inside the CXL memory controller. The approach yields substantial end-to-end speedups and energy savings over passive CXL memory, including notable improvements for OLAP, KVStore, LLM, DLRM, and graph analytics, while maintaining compatibility with unmodified CXL.mem. The results show near-linear scaling across multiple memories and promising cost efficiency, supporting wide applicability in memory-bound, data-centric workloads.

Abstract

Emerging Compute Express Link (CXL) enables cost-efficient memory expansion beyond the local DRAM of processors. While its CXL$.$mem protocol provides minimal latency overhead through an optimized protocol stack, frequent CXL memory accesses can result in significant slowdowns for memory-bound applications whether they are latency-sensitive or bandwidth-intensive. The near-data processing (NDP) in the CXL controller promises to overcome such limitations of passive CXL memory. However, prior work on NDP in CXL memory proposes application-specific units that are not suitable for practical CXL memory-based systems that should support various applications. On the other hand, existing CPU or GPU cores are not cost-effective for NDP because they are not optimized for memory-bound applications. In addition, the communication between the host processor and CXL controller for NDP offloading should achieve low latency, but existing CXL$.$io/PCIe-based mechanisms incur $μ$s-scale latency and are not suitable for fine-grained NDP. To achieve high-performance NDP end-to-end, we propose a low-overhead general-purpose NDP architecture for CXL memory referred to as Memory-Mapped NDP (M$^2$NDP), which comprises memory-mapped functions (M$^2$func) and memory-mapped $μ$threading (M$^2μ$thread). M$^2$func is a CXL$.$mem-compatible low-overhead communication mechanism between the host processor and NDP controller in CXL memory. M$^2μ$thread enables low-cost, general-purpose NDP unit design by introducing lightweight $μ$threads that support highly concurrent execution of kernels with minimal resource wastage. Combining them, M$^2$NDP achieves significant speedups for various workloads by up to 128x (14.5x overall) and reduces energy by up to 87.9% (80.3% overall) compared to baseline CPU/GPU hosts with passive CXL memory.

Figures (15)

• Figure 1: (a) Roofline analysis of workload performance with data in local memory vs. CXL memory. (b) Impact of Load-to-Use (LtU) latencies of local and CXL memories on the 95th percentile (P95) latency of key-value store ([1.0]KVS_A). CXL memory latency can vary depending on the implementation cxl_uiuccmm-bpond. Evaluation methodology is described in § \ref{['sec:methodology']}.
• Figure 2: CXL implementation and measured round-trip latencies for CXL.mem (figure and numbers adapted from D. D. Sharma cxl_scale). CXL.$Mem refers to both CXL.cache and CXL.mem (TL: transaction layer, LL: link layer).
• Figure 3: Overview of the proposed system with M$^2$NDP-enabled CXL memory.
• Figure 4: Example NDP kernel launch using M$^2$func with VectorAdd NDP kernel that computes C=A+B. Vectors A, B, and C are placed at 0xA000, 0xB000, and 0xC000, respectively. It is assumed that the virtual address 0x60040 is translated into physical address 0x10040. Each $\mu$thread computes a 32B (8x4B) partial vector output. Other datapath components are not shown.
• Figure 5: Example timelines with different NDP offloading schemes. One-way latencies of CXL.mem, CXL.io, and kernel execution are parameterized as $x$, $y$, and $z$, respectively. Their known minimal values are $x$=$\sim$75 ns from 150 ns load-to-use latency for CXL memory cxl_scalepond, $y$=$\sim$500 ns from $\sim$1 $\mu$s DMA pcie-latency. An example value for $z$ is 6.4 $\mu$s NDP kernel runtime from [1.0]DLRM(SLS)-B32 (§ \ref{['sec:perf']}). For M$^2$func, we assume a synchronous launch while also showing the arrow for an alternative asynchronous launch. For the ring buffer, CMD and CMP refer to command and completion messages enqueued into the ring buffers, respectively. Two pairs of CMD and CMP are needed for kernel launch and error checks cuda_error_check. While the barrier for M$^2$func overlaps with the kernel, the one needed for ring buffer is in the critical path.