Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models

Zongkai Liu, Fanqing Meng, Lingxiao Du, Zhixiang Zhou, Chao Yu, Wenqi Shao, Qiaosheng Zhang

TL;DR

This work tackles training instability in rule-based reinforcement learning for language models by identifying the detrimental role of importance-sampling ratios in loss formulations. It introduces Clipped Policy Gradient Optimization with Policy Drift (CPGD), a ratio-free policy-gradient objective that includes a log-ratio clip and a forward KL-based policy-drift penalty to ensure proximal updates and stability. The authors provide convergence guarantees and demonstrate across six multimodal math benchmarks that CPGD yields superior stability and performance, including an overall improvement of $+11.0\%$ and notable gains on MMK12, MathVista, and MathVision, while maintaining practical implementability. The approach offers a robust, scalable RL solution for post-training LM optimization with rule-based rewards and is accompanied by an open-source implementation.

Abstract

Recent advances in rule-based reinforcement learning (RL) have significantly improved the reasoning capability of language models (LMs) with rule-based rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO -- often suffer from training instability, where large policy updates and improper clipping can lead to training collapse. To address this issue, we propose Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel algorithm designed to stabilize policy learning in LMs. CPGD introduces a policy drift constraint based on KL divergence to dynamically regularize policy updates, and leverages a clip mechanism on the logarithm of the ratio to prevent excessive policy updates. We provide theoretical justification for CPGD and demonstrate through empirical analysis that it mitigates the instability observed in prior approaches. Furthermore, we show that CPGD significantly improves performance while maintaining training stability. Our implementation balances theoretical rigor with practical usability, offering a robust alternative for RL in the post-training of LMs. We release our code at https://github.com/ModalMinds/MM-EUREKA.

CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models

TL;DR

This work tackles training instability in rule-based reinforcement learning for language models by identifying the detrimental role of importance-sampling ratios in loss formulations. It introduces Clipped Policy Gradient Optimization with Policy Drift (CPGD), a ratio-free policy-gradient objective that includes a log-ratio clip and a forward KL-based policy-drift penalty to ensure proximal updates and stability. The authors provide convergence guarantees and demonstrate across six multimodal math benchmarks that CPGD yields superior stability and performance, including an overall improvement of and notable gains on MMK12, MathVista, and MathVision, while maintaining practical implementability. The approach offers a robust, scalable RL solution for post-training LM optimization with rule-based rewards and is accompanied by an open-source implementation.

Abstract

Recent advances in rule-based reinforcement learning (RL) have significantly improved the reasoning capability of language models (LMs) with rule-based rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO -- often suffer from training instability, where large policy updates and improper clipping can lead to training collapse. To address this issue, we propose Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel algorithm designed to stabilize policy learning in LMs. CPGD introduces a policy drift constraint based on KL divergence to dynamically regularize policy updates, and leverages a clip mechanism on the logarithm of the ratio to prevent excessive policy updates. We provide theoretical justification for CPGD and demonstrate through empirical analysis that it mitigates the instability observed in prior approaches. Furthermore, we show that CPGD significantly improves performance while maintaining training stability. Our implementation balances theoretical rigor with practical usability, offering a robust alternative for RL in the post-training of LMs. We release our code at https://github.com/ModalMinds/MM-EUREKA.
Paper Structure (23 sections, 4 theorems, 27 equations, 1 figure, 3 tables)

This paper contains 23 sections, 4 theorems, 27 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Problem formulation
  5. Rule-based reinforcement learning
  6. The proposed method
  7. Clipped Policy Gradient Optimization with Policy Drift (CPGD)
  8. Training collapse
  9. Implementation
  10. Experiments
  11. Experiments setup
  12. Main results
  13. Ablation study
  14. Discussion
  15. Importance sampling
  16. ...and 8 more sections

Key Result

Proposition 1

Let $\theta_0$ be a parameter such that the importance-sampling ratio satisfies $|\frac{\pi_{\theta_0}(\mathbf{y}|\mathbf{x})}{\pi_{\theta_{old}}(\mathbf{y}|\mathbf{x})} - 1|= \epsilon$. Consider updating $\theta_0$ using either (i) the PPO-clip objective, resulting in parameter $\theta_1^{\text{PPO After one update step, both PPO and CPG increase the importance-sampling ratio deviation from the o

Figures (1)

  • Figure 1: Accuracy, clipping fraction and response length curves throughout training.

Theorems & Definitions (6)

  • Proposition 1
  • Theorem 1
  • Proposition 2
  • proof
  • Theorem 2
  • proof