PRISM: Probabilistic Runtime Insights and Scalable Performance Modeling for Large-Scale Distributed Training

TL;DR

The paper addresses the unpredictability of training times in very large-scale distributed GPU deployments. It introduces PRISM, a probabilistic runtime insights framework that uses operator-level latency distributions and DAG-based modeling to forecast end-to-end training time with probabilistic guarantees, validated against real-system traces. Key findings include that communication kernels, especially AllGather and ReduceScatter, disproportionately affect variability, that slow-node placement can drastically alter step time, and that cross-datacenter network variability can significantly degrade performance unless bandwidth is high enough. The work provides a practical, scalable method to guide design decisions for variability-aware training systems and future infrastructure planning across datacenters.

Abstract

Large model training beyond tens of thousands of GPUs is an uncharted territory. At such scales, disruptions to the training process are not a matter of if, but a matter of when -- a stochastic process degrading training productivity. Dynamic runtime variation will become increasingly more frequent as training scales up and GPUs are operated in increasingly power-limited and thermally-stressed environments. At the 64k GPU scale, we already observed 9% GPU time variability for frontier foundation model training. To understand potential causes of variability, we analyze GPU microbenchmarks at scale across a variety of platforms, showing up to 14% variation in GPU performance on GEMM workloads depending on training hardware and deployed environment. Motivated by our analysis and the large design space around performance variability, we present PRISM -- a performance modeling framework that considers the stochastic nature of the large-scale distributed training. The core of PRISM is the statistical method that provides a quantifiable measure for probabilistic guarantees on training time. Using PRISM, we explore the design and optimization space of distributed training, from parallelization methods to next-generation training systems. PRISM is validated with real-system measurement, showing training time prediction accuracy with 20.8% Kolmogorov-Smirnov distance. Using PRISM, we demonstrate that, depending on computation node placement, up to 1.26x performance improvement potential is available if we factor in sensitivities of parallelization strategies to variation. In addition, we use PRISM to identify kernels to optimize for reducing performance variability and predict probability of slow-down for large-scale jobs where variation is magnified. We find optimizing communication kernels, such as AllGather and ReduceScatter, contribute most to minimizing variability in training step time.

Figures (13)

• Figure 1: Distribution of compute and communication time for two LLM training experiments in real-world production training environment -- across 16K and 64K+ GPUs respectively. We find the 64K+ job exhibits 2.03$\times$ higher standard deviation in compute and 2.91$\times$ higher standard deviation in communication operations compared to the 16K job, highlighting increased performance variability at larger scales.
• Figure 2: Taxonomy of Latency Variation. There are a multitude of compute and communication factors that contribute to variability.
• Figure 3: Significant latency variability is observed for (1) GPUs over the fleet (spatial variability) and (2) execution instances of the same computation kernel on the same GPUs (temporal variability).
• Figure 4: A significant source of computation latency variability comes from the dynamically varying frequency setting of GPUs.
• Figure 5: Communication primitive distributions. Pytorch communication collective AllReduce profiled on 1 and 8 8xH200 GPU nodes across both intra-node NVLink (left), and inter-node backend (EFA) network (right), respectively. Intra-node communication is nearly deterministic at large message sizes, though we do measure rare stragglers at small message sizes. In contrast, inter-node traffic is more variable---latency fluctuates at a millisecond level and tail latency can span up to an order of magnitude.