FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management

TL;DR

The paper tackles the costly recovery bottleneck in large-scale LLM training by introducing FFTrainer, a system that enables instant checkpointing and fast failover through state-size reduction, decoupled network initialization, and a lightweight network stack (LCCL) that leverages idle bandwidth. It integrates with PyTorch, Megatron, and DeepSpeed, and demonstrates up to ~98% reduction in recovery time with minimal impact on normal training on a 128-GPU cluster. Key contributions include the checkpoint razor for reducing state size, neighboring redundancy and lazy backup for fast, consistent ckpting, and latency-aware load planning and network state initialization. The approach yields high recovery reliability, scalable performance, and practical applicability to future, larger, and faster clusters, with future work on further compression and optimization.

Abstract

Recent developments in large language models (LLMs) have introduced new requirements for efficient and robust training. As LLM clusters scale, node failures, lengthy recoveries, and bulky checkpoints erode efficiency. Infrequent asynchronous checkpoints trigger costly rollbacks, yet higher frequencies add prohibitive overhead. To address these challenges, we propose FFTrainer, a system designed for robust LLM training. FFTrainer leverages surplus network capacity to quickly save and load states, thereby preventing rollbacks and accelerating recovery. Compared with prior checkpointing approaches, FFTrainer reduces recovery time by up to 98% and mitigates GPU utilization loss by up to 68% without hindering normal training.

Figures (10)

• Figure 1: FFTrainer recovery timeline vs. PyTorch solutions.
• Figure 2: FFTrainer communication among workers with role $\{r_d,r_p,r_t\}$. Each iteration: workers send heatbeats to the controller (①) and backup their states on the neighbour in the data parallel group (③). The controller spreads data indexing to the rank 0 in each tensor parallel group (②). When pod 2 fails, the controller can detect failure via heartbeats and forward states from pod 1 to the substitute (④).
• Figure 3: FFTrainer checkpointing approach vs. current solutions. FFTrainer introduces no blocking time between iterations but overlaps all checkpointing steps with computation.
• Figure 4: Checkpointing overhead for each system. The hollow part represents the difference of training time with/without CKPT, i.e., overhead of asynchronous checkpointing.
• Figure 5: Relative MFU loss for failures at different MTBF.