Role-Based Fault Tolerance System for LLM RL Post-Training

TL;DR

The paper addresses faults in RL post-training, where rollout and training interleave and both phases are susceptible to failures. It proposes RobustRL, a role-based fault-tolerance framework that detects, restarts, and reconnects failed roles (trainer or rollout) while preserving the progress of other roles. Key contributions include phase-aware fault detection, decoupled trainer and rollout recovery (with rollout warm standby), and UCX-based dynamic weight synchronization, all implemented on Verl with ByteCheckpoint support. Empirical results on large GPU clusters show RobustRL achieves around 80% ETTR and 8.4–17.4% faster end-to-end training than ByteRobust, validating its effectiveness in reducing restart overhead and improving training throughput in fault-prone RL post-training tasks.

Abstract

RL post-training for LLMs has been widely scaled to enhance reasoning and tool-using capabilities. However, RL post-training interleaves training and inference workloads, exposing the system to faults from both sides. Existing fault tolerance frameworks for LLMs target either training or inference, leaving the optimization potential in the asynchronous execution unexplored for RL. Our key insight is role-based fault isolation so the failure in one machine does not affect the others. We treat trainer, rollout, and other management roles in RL training as distinct distributed sub-tasks. Instead of restarting the entire RL task in ByteRobust, we recover only the failed role and reconnect it to living ones, thereby eliminating the full-restart overhead including rollout replay and initialization delay. We present RobustRL, the first comprehensive robust system to handle GPU machine errors for RL post-training Effective Training Time Ratio improvement. (1) \textit{Detect}. We implement role-aware monitoring to distinguish actual failures from role-specific behaviors to avoid the false positive and delayed detection. (2) \textit{Restart}. For trainers, we implement a non-disruptive recovery where rollouts persist state and continue trajectory generation, while the trainer is rapidly restored via rollout warm standbys. For rollout, we perform isolated machine replacement without interrupting the RL task. (3) \textit{Reconnect}. We replace static collective communication with dynamic, UCX-based (Unified Communication X) point-to-point communication, enabling immediate weight synchronization between recovered roles. In an RL training task on a 256-GPU cluster with Qwen3-8B-Math workload under 10\% failure injection frequency, RobustRL can achieve an ETTR of over 80\% compared with the 60\% in ByteRobust and achieves 8.4\%-17.4\% faster in end-to-end training time.

Figures (19)

• Figure 1: Different RL training architectures. The GPU role includes trainer, rollout and hybrid (colocate the both).
• Figure 2: Fault detection of pre-train applied in RL. (a) Rank-level leads to false positive. (b) Cluster-level leads to delay.
• Figure 3: Trajectory length distribution (a) and time cost of each step (b) of Search R1 search_r1 training in HotPotQA dataset HotpotQA with Qwen2.5-7B qwen2.5.
• Figure 4: System overview of RobustRL.
• Figure 5: Role- and phase-aware fault detection in semi-sync RL training. Row 2 is the hybrid. TPS: Throughput Per Second.