Training LLMs with Fault Tolerant HSDP on 100,000 GPUs

TL;DR

This work tackles the challenge of training ultra-large language models on 100K GPUs where synchronous training incurs prohibitive stall times due to frequent failures. It introduces Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP), a CPU–GPU hybrid framework that uses replica-level data parallelism and a Fault Tolerant All Reduce (FTAR) to maintain progress when failures occur, plus a non-blocking catch-up protocol to minimize stalls. Empirical results show that FT-HSDP reduces recovery stall from $10$ minutes to $3$ minutes and raises effective training time from $44\%$ to $80\%$ at scale, while maintaining model accuracy comparable to synchronous baselines; even in aggressive fault scenarios, final results remain robust, with learning-rate interventions helping stabilize fluctuations. The paper also details scalable network design, fault diagnosis tools, and large-scale CPU emulation to validate the approach before hardware is available, underscoring the practical impact for future trillion-token pretraining on vast GPU clusters.

Abstract

Large-scale training systems typically use synchronous training, requiring all GPUs to be healthy simultaneously. In our experience training on O(100K) GPUs, synchronous training results in a low efficiency due to frequent failures and long recovery time. To address this problem, we propose a novel training paradigm, Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP). FT-HSDP uses data parallel replicas as units of fault tolerance. When failures occur, only a single data-parallel replica containing the failed GPU or server is taken offline and restarted, while the other replicas continue training. To realize this idea at scale, FT-HSDP incorporates several techniques: 1) We introduce a Fault Tolerant All Reduce (FTAR) protocol for gradient exchange across data parallel replicas. FTAR relies on the CPU to drive the complex control logic for tasks like adding or removing participants dynamically, and relies on GPU to perform data transfer for best performance. 2) We introduce a non-blocking catch-up protocol, allowing a recovering replica to join training with minimal stall. Compared with fully synchronous training at O(100K) GPUs, FT-HSDP can reduce the stall time due to failure recovery from 10 minutes to 3 minutes, increasing effective training time from 44\% to 80\%. We further demonstrate that FT-HSDP's asynchronous recovery does not bring any meaning degradation to the accuracy of the result model.

Figures (9)

• Figure 1: Network architecture of the training cluster.
• Figure 2: Breakdown of recovery time as the scale increases.
• Figure 3: Architecture of FT-HSDP.
• Figure 4: Design of Rank 0 (leader) in FT-HSDP.
• Figure 5: Details of FTAR protocol.