LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms

TL;DR

This work tackles the challenge of diagnosing performance issues in production multi-tenant LLM training platforms under black-box constraints. It introduces LLMPrism, a non-intrusive framework that reconstructs per-GPU training timelines from network flow data through four phases: training job recognition, parallelism identification, timeline reconstruction, and performance diagnosis. Empirical results on Platform-X show near-perfect accuracy in job and parallelism recognition and timeline reconstruction errors below $0.3\%$, enabling timely detection of failures and network bottlenecks. The approach demonstrates practical impact and generalizability for improving the reliability and efficiency of large-scale LLM training in real-world, privacy-preserving environments.

Abstract

Large Language Models (LLMs) have brought about revolutionary changes in diverse fields, rendering LLM training of utmost importance for modern enterprises. To meet this demand, multi-tenant large-scale LLM training platforms have been built to offer LLM training services. Nevertheless, due to the complexity and synchronous nature of LLM training process, performance issues occur frequently and can result in substantial resource wastage. The limited visibility from the perspective of platform providers impedes existing profiling methods and poses challenges to the monitoring and diagnosis of the performance of LLM training jobs. For the first time, this paper proposes the utilization of underlying network flow data to reconstruct the training timelines of jobs based on the distinct characteristics in the LLM training procedure. We design LLMPrism, the first black-box performance diagnosis system for LLM training platforms. By progressively recognizing LLM training jobs, identifying their parallelism strategies, and reconstructing the training timelines, LLMPrism achieves non-intrusive, lightweight, and continuous monitoring of LLM training systems. Leveraging this monitoring capability, it further effectively diagnoses potential performance issues. Since Oct. 2024, LLMPrism has been deployed on our large-scale production Platform-X, in which the evaluations and deployment experiences demonstrate that LLMPrism can achieve accurate timeline reconstruction with an error within 0.3% and effectively diagnose various performance issues.

Figures (5)

• Figure 1: The training parallelism and network monitoring.
• Figure 2: The overall framework of LLMPrism.
• Figure 3: LLM training job recognition procedure of LLMPrism in training clusters with 2,880 GPUs in Platform-X.
• Figure 4: An example of reconstructed training timeline.
• Figure 5: Switch-level diagnosis of LLMPrism in Platform-X using one-hour aggregated average bandwidths.