QuRL: Efficient Reinforcement Learning with Quantized Rollout

TL;DR

This work proposes Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout, and identifies the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.

Figures (12)

• Figure 1: Overview of QuRL training. The sampling model $\theta_{\mathrm{old}}$ is quantized to $\hat{\theta}_{\mathrm{old}}$ for rollout.
• Figure 1: Comparison of GSM8k accuracy.
• Figure 2: Comparison of (a) training rewards and (b) token clipped fraction under different training objective or quantization.
• Figure 2: Comparison of AIME 2024 accuracy.
• Figure 3: Training dynamics of QuRL. (a) Training collapses after 1000 steps due to increased KL divergence between behavior and proximal policy, and (b) the maximum value of the proximal-to-behavior policy ratio.