Laminar: A Scalable Asynchronous RL Post-Training Framework

TL;DR

This work addresses the bottleneck of long-tail trajectory generation in RL post-training for LLMs by introducing trajectory-level asynchrony. Laminar provides a fully decoupled architecture with relay-based asynchronous weight synchronization and a dynamic repack mechanism that consolidates tail trajectories to maximize throughput while preserving convergence. The approach yields up to 5.48x end-to-end throughput improvements on large-scale clusters and demonstrates robust fault tolerance for long-running jobs. The results indicate substantial practical impact for scalable RL post-training of LLMs, enabling more efficient reasoning enhancement at scale.

Abstract

Reinforcement learning (RL) post-training for Large Language Models (LLMs) is now scaling to large clusters and running for extended durations to enhance model reasoning performance. However, the scalability of existing RL frameworks is limited, as extreme long-tail skewness in RL trajectory generation causes severe GPU underutilization. Current asynchronous RL systems attempt to mitigate this, but they rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule. This global synchronization is ill-suited for the highly skewed and evolving distribution of trajectory generation latency in RL training, crippling training efficiency. Our key insight is that efficient scaling requires breaking this lockstep through trajectory-level asynchrony, which generates and consumes each trajectory independently. We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture. First, we replace global updates with a tier of relay workers acting as a distributed parameter service. This enables asynchronous and fine-grained weight synchronization, allowing rollouts to pull the latest weight anytime without stalling the actor's training loop. Second, a dynamic repack mechanism consolidates long-tail trajectories onto a few dedicated rollouts, maximizing generation throughput. The fully decoupled design also isolates failures, ensuring robustness for long-running jobs. Our evaluation on a 1024-GPU cluster shows that Laminar achieves up to 5.48$\times$ training throughput speedup over state-of-the-art systems, while reducing model convergence time.

Figures (18)

• Figure 1: RL post-training workflow and time breakdown of single-turn (math) dapo and multi-turn (code) tasks swe-bench.
• Figure 2: Trajectory length distribution on AIME dataset and execution latency distribution of code sandbox AIMEretool.
• Figure 3: Comparison of RL systems. (a)-(d) adopt GPU-direct communication to perform global weight synchronization, while (c) needs to copy the previous weight version to dedicate memory buffers for later synchronization. (e) performs asynchronous weight synchronization through RDMA on CPU. $\text{W}_\text{n}$ is the actor weight version at iteration n.
• Figure 4: One-step decode latency of Qwen2.5-7B/32B on H800 GPUs, under various tensor parallel (TP) sizes with decode batch sizes up to the KVCache limit.
• Figure 5: Laminar architecture and training workflow.