PAL: A Variability-Aware Policy for Scheduling ML Workloads in GPU Clusters

TL;DR

This work addresses performance variability in GPU-dense ML clusters by proposing variability-aware scheduling policies that exploit app-specific profiles. The authors introduce PM-First and PAL, where offline application classification and per-GPU variability measurements drive placement decisions; PAL additionally balances variability with network locality using an efficient LxV traversal approach. Across real Frontera hardware and trace-driven simulations, PAL delivers substantial gains in average job completion time, makespan, and cluster utilization compared with state-of-the-art, variability-agnostic baselines. The results demonstrate that incorporating variability into placement, and jointly optimizing locality, yields scalable, practical improvements for large GPU clusters powering ML workloads, with clear implications for HPC+ML environments.

Abstract

Large-scale computing systems are increasingly using accelerators such as GPUs to enable peta- and exa-scale levels of compute to meet the needs of Machine Learning (ML) and scientific computing applications. Given the widespread and growing use of ML, including in some scientific applications, optimizing these clusters for ML workloads is particularly important. However, recent work has demonstrated that accelerators in these clusters can suffer from performance variability and this variability can lead to resource under-utilization and load imbalance. In this work we focus on how clusters schedulers, which are used to share accelerator-rich clusters across many concurrent ML jobs, can embrace performance variability to mitigate its effects. Our key insight to address this challenge is to characterize which applications are more likely to suffer from performance variability and take that into account while placing jobs on the cluster. We design a novel cluster scheduler, PAL, which uses performance variability measurements and application-specific profiles to improve job performance and resource utilization. PAL also balances performance variability with locality to ensure jobs are spread across as few nodes as possible. Overall, PAL significantly improves GPU-rich cluster scheduling: across traces for six ML workload applications spanning image, language, and vision models with a variety of variability profiles, PAL improves geomean job completion time by 42%, cluster utilization by 28%, and makespan by 47% over existing state-of-the-art schedulers.

Figures (20)

• Figure 1: Modular view of Blox job scheduling.
• Figure 2: Variability-aware scheduling overview.
• Figure 3: Classification of applications using 2-dimensional clustering over the $\text{Util}_{\text{DRAM}} \times \max(\text{Util}_{\text{FU}})$ space.
• Figure 4: PM-First Placement Policy with job queue reordering based on placement priority.
• Figure 5: Example showing clustering on a 128-GPU cluster for a class A application. A blue cross marks each bin's centroid.