Longitudinal Analysis of GPU Workloads on Perlmutter

TL;DR

This work analyzes GPU workloads on the Perlmutter supercomputer by leveraging GPU-level time-series counters collected with LDMS/DCGM and linked to Slurm metadata. It introduces and applies spatial and temporal analysis methods, including a time-windowed spatial imbalance metric, temporal imbalance, and burstiness to characterize per-job GPU usage. The study reveals notable spatial and temporal imbalances, diverse utilization patterns across job types and GPU cores, and strong relationships between compute activity, data transfer, and overall GPUUtilization, with ML workloads showing distinct patterns. These findings offer actionable guidance for workload optimization and future HPC system design, and point to extensions into anomaly detection and energy-aware analyses.

Abstract

GPGPU-based clusters and supercomputers have become extremely popular in the last ten years. There is a large number of GPGPU hardware counters exposed to the users, however, very little analysis has been done regarding insights they might offer about workloads running on them. In this work, we address this gap by analyzing previously unexplored GPU hardware counters collected via Lightweight Distributed Metric Service on Perlmutter, a leadership-class supercomputer. We examine several hardware counters related to utilization of GPU cores and memory and present a detailed spatial and temporal analysis of GPU workloads. We investigate spatial imbalance -- uneven GPU usage across multiple GPUs within a job. Our temporal study examines how GPU usage fluctuates during a job's lifetime, introducing two new metrics -- burstiness (the irregularity of large utilization changes) and temporal imbalance (deviations from mean utilization over time). Additionally, we compare machine learning and traditional high performance computing jobs. Our findings uncover inefficiencies and imbalances that can inform workload optimization and future HPC system design.

Figures (13)

• Figure 1: The plots show the number of jobs with CDF, duration in hours (log-scale), and mean of GPU_UTIL by the number of GPUs (left to right). Red dots on boxplots indicate mean values. Most jobs use a single node, durations vary widely but average 1-3 hours for most job sizes, and larger jobs tend to have lower mean GPU utilization.
• Figure 2: The plot demonstrates the number and percentage of jobs using different combinations of GPU cores. FP64 is the most commonly used core, FP16 is rarely used, and tensor cores are often utilized alongside FP64 and FP32.
• Figure 3: The plots show the number of jobs (top) and GPU hours in log-scale (bottom) by mean of GPU_UTIL with CDF. The red dots on the boxplots indicate the mean values. Mean of GPU_UTIL varies greatly. 43% of jobs fall within the low utilization range (0-30%).
• Figure 4: The plots show the distribution of spatial imbalance of GPU_UTIL for jobs grouped by mean of GPU_UTIL ranges (0-30%, 31-69%, 70-100%, left to right). Low-utilization jobs exhibit the highest spatial imbalance. 97.6% of high-utilization have below 0.5 imbalance.
• Figure 5: The plots compare the number (top) and GPU hours (bottom) of ML and non-ML jobs by mean of GPU_UTIL. ML jobs dominate high (70-100%) and the low (0, 9%) utilization bins. They account for more GPU hours. Absolute values are annotated above bars.