Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning

Batuhan K. Karaman, Aditya Rawal, Suhaila Shakiah, Mohammad Ghavamzadeh, Mingyi Hong, Arijit Biswas, Ruida Zhou

TL;DR

DISPO is introduced, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes.

Abstract

Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights >1 increase the average token entropy (i.e., exploration) while weights <1 decrease it (i.e., distillation) -- both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights >1) or vanishing response lengths (when weights <1). By separately tuning these four clipping parameters, DISPO maintains the exploration-distillation balance while preventing catastrophic failures, achieving 61.04% on AIME'24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various benchmarks and models.

DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning

TL;DR

DISPO is introduced, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes.

Abstract

Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights >1 increase the average token entropy (i.e., exploration) while weights <1 decrease it (i.e., distillation) -- both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights >1) or vanishing response lengths (when weights <1). By separately tuning these four clipping parameters, DISPO maintains the exploration-distillation balance while preventing catastrophic failures, achieving 61.04% on AIME'24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various benchmarks and models.
Paper Structure (37 sections, 12 equations, 17 figures, 2 tables)

This paper contains 37 sections, 12 equations, 17 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Outline
  3. Related Work
  4. Foundations of Policy Gradient Methods
  5. RL for Language Models
  6. Background
  7. GRPO Algorithm
  8. DAPO Algorithm
  9. REINFORCE Algorithm
  10. CISPO Algorithm
  11. Decoupled Importance Sampling-weighted Policy Optimization (DISPO)
  12. Gradient-weight view
  13. Analyzing the Policy Update Regimes
  14. Regime 1: Amplified Positive Updates ($\hat{A}_{i,t} > 0$, $r_{i,t}(\theta) > 1$)
  15. Regime 2: Suppressed Positive Updates ($\hat{A}_{i,t} > 0$, $r_{i,t}(\theta) < 1$)
  16. ...and 22 more sections

Figures (17)

  • Figure 1: Learning curves of RLVR algorithms.
  • Figure 2: DISPO extends REINFORCE with (i) group-relative advantage estimation, (ii) token-level normalization, and (iii) decoupled IS weight $r_{i,t}^d(\theta)$. Each $\epsilon$ in the decoupled IS weight controls a distinct policy update regime.
  • Figure 3: Gradient weight $w_{i,t}(\theta)$ as a function of the importance-sampling weight $r_{i,t}(\theta)$.
  • Figure 4: DISPO's four policy update regimes.
  • Figure 5: Accuracy and entropy curves of DAPO, CISPO, and DISPO.
  • ...and 12 more figures