MPI-over-CXL: Enhancing Communication Efficiency in Distributed HPC Systems

TL;DR

This work addresses the high cost of data movement in traditional MPI implementations on distributed HPC systems. It introduces MPI-over-CXL, a cache-coherent shared-memory MPI paradigm that leverages CXL 3.x to map a unified memory space across multiple hosts and enable pointer-based communication, eliminating expensive data copies. Key contributions include the design of shared memory-backed message queues, unified memory management with memory expanders, and synchronization mechanisms, plus a hardware-software prototype with a CXL 3.2 controller and FPGA-based multi-host emulation. Empirical results show end-to-end speedups up to $1.6\times$ and a reduction in communication time by $8.4\times$, with strong scaling gains up to $2.25\times$ on large node counts, indicating significant potential for improving HPC efficiency and scalability in future clusters.

Abstract

MPI implementations commonly rely on explicit memory-copy operations, incurring overhead from redundant data movement and buffer management. This overhead notably impacts HPC workloads involving intensive inter-processor communication. In response, we introduce MPI-over-CXL, a novel MPI communication paradigm leveraging CXL, which provides cache-coherent shared memory across multiple hosts. MPI-over-CXL replaces traditional data-copy methods with direct shared memory access, significantly reducing communication latency and memory bandwidth usage. By mapping shared memory regions directly into the virtual address spaces of MPI processes, our design enables efficient pointer-based communication, eliminating redundant copying operations. To validate this approach, we implement a comprehensive hardware and software environment, including a custom CXL 3.2 controller, FPGA-based multi-host emulation, and dedicated software stack. Our evaluations using representative benchmarks demonstrate substantial performance improvements over conventional MPI systems, underscoring MPI-over-CXL's potential to enhance efficiency and scalability in large-scale HPC environments.

Figures (7)

• Figure 1: Traditional MPI and emerging CXL.
• Figure 2: Comparison of MPI communication paradigms.
• Figure 3: Memory management and data structures.
• Figure 4: Implementation of MPI asynchronous operations.
• Figure 5: Evaluation environment.