Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling All-to-all Operations Across Emerging Many-Core Supercomputers

Shannon Kinkead, Jackson Wesley, Whit Schonbein, David DeBonis, Matthew G. F. Dosanjh, Amanda Bienz

TL;DR

This work tackles the scalability of all-to-all collectives on emerging many-core supercomputers with multi-socket and NUMA architectures. It introduces hierarchical, node-aware, locality-aware, and a novel multi-leader with node-aware all-to-all algorithms, evaluating them on three Sapphire Rapids/MI300A-based systems. The study demonstrates that multi-leader node-aware excels for small messages and that locality-aware approaches can outperform standard node-aware strategies for large data when locality groups align with hardware regions. The results inform algorithm selection and point toward dynamic auto-tuning to optimize all-to-all performance across architectures and workloads.

Abstract

Performant all-to-all collective operations in MPI are critical to fast Fourier transforms, transposition, and machine learning applications. There are many existing implementations for all-to-all exchanges on emerging systems, with the achieved performance dependent on many factors, including message size, process count, architecture, and parallel system partition. This paper presents novel all-to-all algorithms for emerging many-core systems. Further, the paper presents a performance analysis against existing algorithms and system MPI, with novel algorithms achieving up to 3x speedup over system MPI at 32 nodes of state-of-the-art Sapphire Rapids systems.

Scaling All-to-all Operations Across Emerging Many-Core Supercomputers

TL;DR

This work tackles the scalability of all-to-all collectives on emerging many-core supercomputers with multi-socket and NUMA architectures. It introduces hierarchical, node-aware, locality-aware, and a novel multi-leader with node-aware all-to-all algorithms, evaluating them on three Sapphire Rapids/MI300A-based systems. The study demonstrates that multi-leader node-aware excels for small messages and that locality-aware approaches can outperform standard node-aware strategies for large data when locality groups align with hardware regions. The results inform algorithm selection and point toward dynamic auto-tuning to optimize all-to-all performance across architectures and workloads.

Abstract

Performant all-to-all collective operations in MPI are critical to fast Fourier transforms, transposition, and machine learning applications. There are many existing implementations for all-to-all exchanges on emerging systems, with the achieved performance dependent on many factors, including message size, process count, architecture, and parallel system partition. This paper presents novel all-to-all algorithms for emerging many-core systems. Further, the paper presents a performance analysis against existing algorithms and system MPI, with novel algorithms achieving up to 3x speedup over system MPI at 32 nodes of state-of-the-art Sapphire Rapids systems.
Paper Structure (13 sections, 18 figures, 1 table)

This paper contains 13 sections, 18 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Methods
  5. Hierarchical
  6. Node-Aware
  7. Combining Multi-leader with Node-Aware
  8. Results
  9. Inter- and Intra-node communication
  10. Amber and Tuolomne
  11. Discussion
  12. Future Work
  13. Conclusions

Figures (18)

  • Figure 1: Alltoall between P processes
  • Figure 2: Hierarchical all-to-all. Blue arrows indicate the intra-node gather, red arrows show the internode all-to-all operation, and yellow arrows indicate the intra-node scatter operation
  • Figure 3: Multi-leader all-to-all, 2 leaders per node. Blue arrows indicate the intra-node gather, red nodes show the all-to-all operation between leaders, and yellow arrows show the intra-node scatter operation.
  • Figure 4: Node-aware all-to-all. Red Arrows indicate the internode all-to-all operation, while blue arrows indicate the intra-node all-to-all operation.
  • Figure 5: Locality-aware all-to-all, 2 groups per node
  • ...and 13 more figures