A Distributed Framework for Causal Modeling of Performance Variability in GPU Traces

TL;DR

This work tackles run-to-run performance variability in GPU traces within heterogeneous HPC systems by introducing an end-to-end distributed framework that scales across many traces. It combines binning-based data reduction, MPI-based parallelism, and causal modeling to expose cause-effect relationships between kernel characteristics, memory behavior, and performance variance, with outputs including causal graphs and parallel coordinate plots. The key contributions are a scalable distributed analysis pipeline, causality-driven insight into variability drivers, and an automated workflow from raw telemetry to actionable performance knowledge. The framework achieves strong scalability, demonstrated by a $67\%$ improvement in handling multiple traces and near-linear speedups as MPI ranks grow, enabling practical time-to-insight for large-scale GPU trace analysis.$

Abstract

Large-scale GPU traces play a critical role in identifying performance bottlenecks within heterogeneous High-Performance Computing (HPC) architectures. However, the sheer volume and complexity of a single trace of data make performance analysis both computationally expensive and time-consuming. To address this challenge, we present an end-to-end parallel performance analysis framework designed to handle multiple large-scale GPU traces efficiently. Our proposed framework partitions and processes trace data concurrently and employs causal graph methods and parallel coordinating chart to expose performance variability and dependencies across execution flows. Experimental results demonstrate a 67% improvement in terms of scalability, highlighting the effectiveness of our pipeline for analyzing multiple traces independently.

Figures (5)

• Figure 1: Overview of the parallel GPU trace analysis framework. The system combines MPI-based distributed parallelism with shared-memory threading to process large GPU activity traces efficiently. MPI ranks divide the workload across nodes (Steps 1–4), each spawning local worker processes to analyze bins in parallel (Steps 5–7). Results are then aggregated (Step 8), and high-variability bins are identified for detailed analysis (Step 9).
• Figure 2: Kernel variability across different ranks and Kernel. Each row illustrates four examples: (a–d) correspond to Ranks 0–1 and (e–h) to Ranks 2–3. Kernel X refers to the associated demangledName, with two kernels randomly selected from different rank datasets. The target is kernel_duration, and perf_variation is computed as $\left| \mathrm{avg}(\texttt{kernel\_duration}) - \texttt{kernel\_duration} \right|$. Black arrows indicate positive influence (e.g., increasing gridX increases variability), while red arrows indicate negative influence on perf_variablity. Edge weight defines change of % of a feature has positive or negative influence on perf_variation
• Figure 3: Overview of kernel variability across ranks and kernels. This figure combines results from all kernels to illustrate variability across different ranks. Due to space constraints, only two representative kernels are randomly selected and shown. The target is kernel_duration, and perf_variation is computed as $\left| \mathrm{avg}(\texttt{kernel\_duration}) - \texttt{kernel\_duration} \right|$. Edge weight defines change of % of a feature has positive or negative influence on perf_variation.
• Figure 4: Parallel coordinates plot illustrating kernel execution characteristics under varying memory access scenarios for the Rank 3 dataset. Each line represents an individual kernel invocation, with axes corresponding to elapsed start time, kernel name, memory copy kind, memory transfer size, kernel duration, and memory stall duration. Line colors encode kernel duration, revealing correlations between transfer size and stall time that highlight performance bottlenecks in memory-bound operations.
• Figure 5: Overview of variability and scalability, illustrating performance behavior across datasets and ranks.