A More Scalable Sparse Dynamic Data Exchange

TL;DR

The work addresses the bottleneck of irregular communication in MPI-based HPC by introducing an MPI extension API (MPI Advance) for sparse dynamic data exchange (SDDE) and two locality-aware SDDE algorithms. It presents API definitions for constant-size and variable-size exchanges, and four implementations (personalized, non-blocking, RMA, and locality-aware) with locality-aware variants showing up to 20x speedups on large-scale tests. The authors implement these methods in MPI Advance and demonstrate scaling benefits on SuiteSparse problems on the Quartz cluster, highlighting reduced inter-region messaging and improved strong scaling. The paper also discusses practical considerations, limitations of RMA for variable sizes, and future directions including dynamic algorithm selection and adaptation to heterogeneous architectures.

Abstract

Parallel architectures are continually increasing in performance and scale, while underlying algorithmic infrastructure often fail to take full advantage of available compute power. Within the context of MPI, irregular communication patterns create bottlenecks in parallel applications. One common bottleneck is the sparse dynamic data exchange, often required when forming communication patterns within applications. There are a large variety of approaches for these dynamic exchanges, with optimizations implemented directly in parallel applications. This paper proposes a novel API within an MPI extension library, allowing for applications to utilize the variety of provided optimizations for sparse dynamic data exchange methods. Further, the paper presents novel locality-aware sparse dynamic data exchange algorithms. Finally, performance results show significant speedups up to 20x with the novel locality-aware algorithms.

Figures (10)

• Figure 1: An example of a $4\times4$ matrix and a vector distributed across 4 processes.
• Figure 2: Data dependency chart when performing the matrix vector product illustrated in Figure \ref{['matrix_partition']}
• Figure 3: MPIX_Alltoall_crs API
• Figure 4: MPIX_Alltoallv_crs API
• Figure 5: The cost of the various MPIX_alltoall_crs methods using Mvapich2.

Theorems & Definitions (1)

• Definition 1: Sparse Dynamic Data Exchange Problem