Fast LLM Post-training via Decoupled and Fastest-of-N Speculation

TL;DR

This paper tackles the bottleneck of rollout in LLM post-training by introducing SpecActor, a system that uses decoupled speculative rollout and Fastest-of-N speculation to accelerate generation without altering training procedures. Decoupled speculation allows drafting and verification to run on separate GPUs, increasing verifier throughput and enabling a higher per-worker verification batch, while a draft window bounds waste from mis-speculation. Fastest-of-N speculation uses an offline draft ladder and dynamic multi-drafter deployment to adaptively select the best draft method for tailed requests, improving acceptance rates and overall speed. Empirical results across production traces and both dense and MoE models show consistent end-to-end speedups of 1.4–2.3x, with rollout speedups of 2.0–2.4x, outperforming baselines like veRL, RLHFuse, and vanilla speculative rollout. SpecActor’s approach scales with model size and training algorithm, offering practical reductions in training time for large-scale RL post-training workflows.

Abstract

Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. This work, SpecActor, achieves fast rollout with speculative decoding that deploys a fast draft path to accelerate the unparallelizable generation, while the correctness is guaranteed by fast parallel verification of the outputs with the original model. SpecActor addresses two foundational challenges that hinder speculation efficiency: (1) a Decoupled speculation method that overcomes the computation inefficiency issue when executing speculative decoding with relative large per-worker batch size -- a common configuration in training but unfriendly to speculation, and (2) a Fastest-of-N speculation method that selects and combines different draft methods according to the rollout progress to approximate the optimal draft method even when the best one is unknown a priori. Extensive evaluations on production traces show that SpecActor accelerates mean rollout speed by 2.0--2.4x, with up to 2.7x speedup, over common post-training baselines. The results are consistent across both dense and MoE models and across different RL algorithms. Notably, SpecActor is 1.1--2.6x faster compared to vanilla speculative rollout in different traces. The accelerated rollout achieves 1.4--2.3x faster end-to-end training time.

Figures (16)

• Figure 1: An illustration of the rollout process in LLM post-training.
• Figure 2: (a) The long rollout in post-training and (b) the training latency of various steps in DAPO-32B-20K training trace (detailed setups described in §\ref{['sec:eval-setup']}).
• Figure 3: (a) An illustration of accelerating post-training via overlapped execution and (b) an illustration of accelerating rollout through scaling to more GPUs.
• Figure 4: An illustration of speculative decoding.
• Figure 5: (a) Distribution of the initial per-worker batch sizes of post-training traces in the last 6 months in a large production cluster. (b) The acceleration of speculative rollout given such a batch size on a Qwen2.5-32B checkpoint.