Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EROICA: Online Performance Troubleshooting for Large-scale Model Training

Yu Guan, Zhiyu Yin, Haoyu Chen, Sheng Cheng, Chaojie Yang, Kun Qian, Tianyin Xu, Pengcheng Zhang, Yang Zhang, Hanyu Zhao, Yong Li, Wei Lin, Dennis Cai, Ennan Zhai

TL;DR

ERCICA is the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems.

Abstract

Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success.

EROICA: Online Performance Troubleshooting for Large-scale Model Training

TL;DR

ERCICA is the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems.

Abstract

Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditional distributed systems or datacenter networks fall short and can hardly apply to real-world training systems. In this paper, we present EROICA, the first online troubleshooting system that provides both fine-grained observation based on profiling, and coverage of all machines in GPU clusters, to diagnose performance issues in production, including both hardware and software problems (or the mixture of both). EROICA effectively summarizes runtime behavior patterns of LMT function executions via online profiling, and leverages differential observability to localize the root cause with minimal production impact. EROICA has been deployed as a production service for large-scale GPU clusters of ~100,000 GPUs for 1.5 years. It has diagnosed a variety of difficult performance issues with 97.5% success.

Paper Structure

This paper contains 24 sections, 11 equations, 23 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Performance Issues in LMT
  4. Common Practices
  5. Goals and Challenges
  6. Main Idea
  7. Eroica Design
  8. Profiling on Performance Degradation
  9. Summarizing Behavior Patterns
  10. Localization
  11. Implementation
  12. Evaluation in Production
  13. Case Study 1: Code-level Issues
  14. Case Study 2: Mixed Code-Hardware Issues
  15. Case Study 3: Diagnose and auto-fix customer-code bugs with AI support
  16. ...and 9 more sections

Figures (23)

  • Figure 1: The full stack of large model training.
  • Figure 2: An overview of LMT performance issues.
  • Figure 3: PCIe bandwidth utilization (from GPU to NIC) during a ring communication without network issues.
  • Figure 4: The ring topology with and without a problematic link. We illustrate a link with only six workers for simplicity. There are 32 in our experiment.
  • Figure 5: GPU-NIC throughput pattern. Left figures are the bandwidth utilizations during the execution of a ring communication function, and right figures show enlarged views of local areas for better observation.
  • ...and 18 more figures