Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Stragglers in Large Model Training Using What-if Analysis

Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi, Wei Jia, Zherui Liu, Shuguang Wang, Haibin Lin, Xin Liu, Aurojit Panda, Jinyang Li

TL;DR

This paper analyzes stragglers in large-scale LLM training using five months of cluster traces and a what-if simulator to contrast real execution with an ideal straggler-free timeline. By constructing OpDuration tensors and dependency models, it quantifies slowdown via $S = T / T_{ideal}$ and GPU-waste, revealing stragglers are prevalent and cause substantial resource loss, with computation being the main contributor. It identifies root causes such as stage partitioning imbalance, sequence-length imbalance, and Python garbage collection pauses, while hardware faults are relatively rare. It also introduces SMon, an online monitoring tool that automates detection and diagnostics, enabling practical mitigation like microbatch balancing and GC scheduling.

Abstract

Large language model (LLM) training is one of the most demanding distributed computations today, often requiring thousands of GPUs with frequent synchronization across machines. Such a workload pattern makes it susceptible to stragglers, where the training can be stalled by few slow workers. At ByteDance we find stragglers are not trivially always caused by hardware failures, but can arise from multiple complex factors. This work aims to present a comprehensive study on the straggler issues in LLM training, using a five-month trace collected from our ByteDance LLM training cluster. The core methodology is what-if analysis that simulates the scenario without any stragglers and contrasts with the actual case. We use this method to study the following questions: (1) how often do stragglers affect training jobs, and what effect do they have on job performance; (2) do stragglers exhibit temporal or spatial patterns; and (3) what are the potential root causes for stragglers?

Understanding Stragglers in Large Model Training Using What-if Analysis

TL;DR

This paper analyzes stragglers in large-scale LLM training using five months of cluster traces and a what-if simulator to contrast real execution with an ideal straggler-free timeline. By constructing OpDuration tensors and dependency models, it quantifies slowdown via and GPU-waste, revealing stragglers are prevalent and cause substantial resource loss, with computation being the main contributor. It identifies root causes such as stage partitioning imbalance, sequence-length imbalance, and Python garbage collection pauses, while hardware faults are relatively rare. It also introduces SMon, an online monitoring tool that automates detection and diagnostics, enabling practical mitigation like microbatch balancing and GC scheduling.

Abstract

Large language model (LLM) training is one of the most demanding distributed computations today, often requiring thousands of GPUs with frequent synchronization across machines. Such a workload pattern makes it susceptible to stragglers, where the training can be stalled by few slow workers. At ByteDance we find stragglers are not trivially always caused by hardware failures, but can arise from multiple complex factors. This work aims to present a comprehensive study on the straggler issues in LLM training, using a five-month trace collected from our ByteDance LLM training cluster. The core methodology is what-if analysis that simulates the scenario without any stragglers and contrasts with the actual case. We use this method to study the following questions: (1) how often do stragglers affect training jobs, and what effect do they have on job performance; (2) do stragglers exhibit temporal or spatial patterns; and (3) what are the potential root causes for stragglers?
Paper Structure (46 sections, 5 equations, 14 figures, 1 table)

This paper contains 46 sections, 5 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Hybrid-Parallel Training and Stragglers
  4. Data parallelism (DP), ZeRO rajbhandari2020zero and fully sharded data parallelism (FSDP zhao2023pytorchfsdp)
  5. Pipeline parallelism (PP).
  6. Tensor parallelism (TP) and context parallelism (CP).
  7. Hybrid parallelism.
  8. The case of straggling for hybrid-parallelism.
  9. Goals and Challenges
  10. Straggler definition.
  11. Our goal.
  12. Methodology
  13. The LLM Training Job Traces
  14. The cluster setup.
  15. Trace collection.
  16. ...and 31 more sections

Figures (14)

  • Figure 1: Rank topology of a DP-PP-TP parallelism.
  • Figure 2: The dependency model used in simulation. "mid" denotes microbatch ID. Operations within a stream execute sequentially. Across streams on the same rank, receive operations (e.g.,$R_{F}$, $Sync_{params}$) precede the corresponding compute operation requiring the data. The opposite dependencies exist for compute and send operations. Across DP ranks, params-sync ($Sync_{params}$) and grads-sync ($Sync_{grads}$) on the same PP rank form a collective group. Across two adjacent PP ranks, their send and receive operations form a P2P pair.
  • Figure 3: CDF of resource waste among all jobs. A given waste percentage's corresponding slowdown ratio is shown in parentheses.
  • Figure 4: CDF of per-step slowdowns normalized by job slowdown for 15 randomly selected steps from each straggling job.
  • Figure 5: Resource wasted due to the slowdown of a single type of operation. Computation slowdown caused the most resource waste, while communication had a minimal impact.
  • ...and 9 more figures