Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ucTrace: A Multi-Layer Profiling Tool for UCX-driven Communication

Emir Gencer, Mohammad Kefah Taha Issa, Ilyas Turimbetov, James D. Trotter, Didem Unat

TL;DR

ucTrace addresses the need for fine-grained UCX-level profiling across CPU-GPU HPC systems by providing multi-layer attribution from UCX transport events to MPI calls and device traffic. The approach instruments UCX at the UCT/UCP layers, attributes communications to MPI and GPU devices via NVIDIA Compute Sanitizer, and offers an interactive visualization suite including timelines, communication matrices, and device graphs. Demonstrations across UCX eager/rendezvous protocols, Open MPI and MPICH AllReduce, a conjugate gradient solver, and GROMACS MD simulations reveal transport usage, topology-dependent behavior, NUMA effects, and potential bottlenecks. While incurring non-trivial runtime overhead, ucTrace delivers actionable insights for administrators and library developers and is designed to be extended to additional libraries and GPU vendors.

Abstract

UCX is a communication framework that enables low-latency, high-bandwidth communication in HPC systems. With its unified API, UCX facilitates efficient data transfers across multi-node CPU-GPU clusters. UCX is widely used as the transport layer for MPI, particularly in GPU-aware implementations. However, existing profiling tools lack fine-grained communication traces at the UCX level, do not capture transport-layer behavior, or are limited to specific MPI implementations. To address these gaps, we introduce ucTrace, a novel profiler that exposes and visualizes UCX-driven communication in HPC environments. ucTrace provides insights into MPI workflows by profiling message passing at the UCX level, linking operations between hosts and devices (e.g., GPUs and NICs) directly to their originating MPI functions. Through interactive visualizations of process- and device-specific interactions, ucTrace helps system administrators, library and application developers optimize performance and debug communication patterns in large-scale workloads. We demonstrate ucTrace's features through a wide range of experiments including MPI point-to-point behavior under different UCX settings, Allreduce comparisons across MPI libraries, communication analysis of a linear solver, NUMA binding effects, and profiling of GROMACS MD simulations with GPU acceleration at scale. ucTrace is publicly available at https://github.com/ParCoreLab/ucTrace.

ucTrace: A Multi-Layer Profiling Tool for UCX-driven Communication

TL;DR

ucTrace addresses the need for fine-grained UCX-level profiling across CPU-GPU HPC systems by providing multi-layer attribution from UCX transport events to MPI calls and device traffic. The approach instruments UCX at the UCT/UCP layers, attributes communications to MPI and GPU devices via NVIDIA Compute Sanitizer, and offers an interactive visualization suite including timelines, communication matrices, and device graphs. Demonstrations across UCX eager/rendezvous protocols, Open MPI and MPICH AllReduce, a conjugate gradient solver, and GROMACS MD simulations reveal transport usage, topology-dependent behavior, NUMA effects, and potential bottlenecks. While incurring non-trivial runtime overhead, ucTrace delivers actionable insights for administrators and library developers and is designed to be extended to additional libraries and GPU vendors.

Abstract

UCX is a communication framework that enables low-latency, high-bandwidth communication in HPC systems. With its unified API, UCX facilitates efficient data transfers across multi-node CPU-GPU clusters. UCX is widely used as the transport layer for MPI, particularly in GPU-aware implementations. However, existing profiling tools lack fine-grained communication traces at the UCX level, do not capture transport-layer behavior, or are limited to specific MPI implementations. To address these gaps, we introduce ucTrace, a novel profiler that exposes and visualizes UCX-driven communication in HPC environments. ucTrace provides insights into MPI workflows by profiling message passing at the UCX level, linking operations between hosts and devices (e.g., GPUs and NICs) directly to their originating MPI functions. Through interactive visualizations of process- and device-specific interactions, ucTrace helps system administrators, library and application developers optimize performance and debug communication patterns in large-scale workloads. We demonstrate ucTrace's features through a wide range of experiments including MPI point-to-point behavior under different UCX settings, Allreduce comparisons across MPI libraries, communication analysis of a linear solver, NUMA binding effects, and profiling of GROMACS MD simulations with GPU acceleration at scale. ucTrace is publicly available at https://github.com/ParCoreLab/ucTrace.
Paper Structure (21 sections, 8 figures, 3 tables)

This paper contains 21 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Related work
  4. Methodology
  5. ucTrace Visualization
  6. Recording UCT Communications
  7. Tracking UCT Completions
  8. UCP Attribution
  9. MPI Attribution
  10. GPU Device Attribution
  11. UCT Active Message Device Attribution
  12. Log Processing
  13. Evaluation
  14. UCX Eager and Rendezvous Protocols
  15. Open MPI and MPICH AllReduce
  16. ...and 6 more sections

Figures (8)

  • Figure 1: UCX architecture
  • Figure 2: Overview of ucTrace and its components.
  • Figure 3: Screenshots of the different features of ucTrace visualizer
  • Figure 4: CUDA-aware MPI_ISend with eager and different rendezvous protocols
  • Figure 5: Communication graphs and communication matrices for MPI_Allreduce on 8 nodes each with 4 cores run with recursive doubling, reduce scatter allgather and ring algorithms using Open MPI v4.1.5 and MPICH v4.3.0 on MareNostrum 5
  • ...and 3 more figures