Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding GPU Triggering APIs for MPI+X Communication

Patrick G. Bridges, Anthony Skjellum, Evan D. Suggs, Derek Schafer, Purushotham V. Bangalore

TL;DR

GPU-triggered MPI communication remains complex and costly in MPI+X ecosystems. The paper provides a taxonomy and classifies nine GPU-triggered MPI proposals (including stream-triggered, kernel-triggered, and GPU-initiated models) with representative implementations from MPICH, HPE, MPI-ACX, and Intel. It analyzes design choices, evaluates API reuse vs creation, initialization vs completion semantics, and progress semantics, and identifies gaps in the MPI standard and proposals. It argues for community bake-offs and converging toward a standard GPU-triggered MPI API for MPI-5 to enable portable, high-performance MPI+GPU communication.

Abstract

GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/graph- and kernel-triggered MPI communication abstractions, whose principal purpose is to enhance the performance of communication when GPU kernels create or consume data for transfer through MPI operations. Researchers and practitioners have proposed multiple potential APIs for stream and/or kernel triggering that span various GPU architectures and approaches, including MPI-4 partitioned point-to-point communication, stream communicators, and explicit MPI stream/queue objects. Designs breaking backward compatibility with MPI are duly noted. Some of these strengthen or weaken the semantics of MPI operations. A key contribution of this paper is to promote community convergence toward a stream- and/or kernel-triggering abstraction by highlighting the common and differing goals and contributions of existing abstractions. We describe the design space in which these abstractions reside, their implicit or explicit use of stream and other non-MPI abstractions, their relationship to partitioned and persistent operations, and discuss their potential for added performance, how usable these abstractions are, and where functional and/or semantic gaps exist. Finally, we provide a taxonomy for stream- and kernel-triggered abstractions, including disambiguation of similar semantic terms, and consider directions for future standardization in MPI-5.

Understanding GPU Triggering APIs for MPI+X Communication

TL;DR

GPU-triggered MPI communication remains complex and costly in MPI+X ecosystems. The paper provides a taxonomy and classifies nine GPU-triggered MPI proposals (including stream-triggered, kernel-triggered, and GPU-initiated models) with representative implementations from MPICH, HPE, MPI-ACX, and Intel. It analyzes design choices, evaluates API reuse vs creation, initialization vs completion semantics, and progress semantics, and identifies gaps in the MPI standard and proposals. It argues for community bake-offs and converging toward a standard GPU-triggered MPI API for MPI-5 to enable portable, high-performance MPI+GPU communication.

Abstract

GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/graph- and kernel-triggered MPI communication abstractions, whose principal purpose is to enhance the performance of communication when GPU kernels create or consume data for transfer through MPI operations. Researchers and practitioners have proposed multiple potential APIs for stream and/or kernel triggering that span various GPU architectures and approaches, including MPI-4 partitioned point-to-point communication, stream communicators, and explicit MPI stream/queue objects. Designs breaking backward compatibility with MPI are duly noted. Some of these strengthen or weaken the semantics of MPI operations. A key contribution of this paper is to promote community convergence toward a stream- and/or kernel-triggering abstraction by highlighting the common and differing goals and contributions of existing abstractions. We describe the design space in which these abstractions reside, their implicit or explicit use of stream and other non-MPI abstractions, their relationship to partitioned and persistent operations, and discuss their potential for added performance, how usable these abstractions are, and where functional and/or semantic gaps exist. Finally, we provide a taxonomy for stream- and kernel-triggered abstractions, including disambiguation of similar semantic terms, and consider directions for future standardization in MPI-5.
Paper Structure (16 sections, 4 figures, 1 table)

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Representative GPU-Triggered Communication Abstractions
  3. MPICH Triggering
  4. HPE One-Sided
  5. Kernel-Triggered Partitioned Communication
  6. Intel GPU-Initiated Communication
  7. A Taxonomy of MPI GPU-Triggered Communication
  8. Classifying Existing MPI GPU Triggering Proposals
  9. API Classification
  10. API Feature Comparisons
  11. Key Gaps
  12. Limitations in the MPI Standard
  13. Limitations across Proposed APIs
  14. Conclusion
  15. Acknowledgments.
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Stream-triggered MPI Ping Pong using MPICH Stream Triggering zhou_mpix_2022
  • Figure 2: Stream-triggered MPI Ping Pong using HPE One-Sided Stream Triggering
  • Figure 3: MPI Ping Pong using MPI-ACX Partitioned Stream Triggering
  • Figure 4: Ping Pong using Intel GPU-Initiated MPI intel-mpi-referenceintel-mpi-gpu-buffers