Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, Tianbao Yang

TL;DR

A new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning, is introduced, which significantly outperforms GRPO and its improved variants such as DAPO, and addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach.

Abstract

The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for an 1.5B model.

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

TL;DR

A new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning, is introduced, which significantly outperforms GRPO and its improved variants such as DAPO, and addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach.

Abstract

The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for an 1.5B model.
Paper Structure (19 sections, 1 theorem, 22 equations, 6 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 1 theorem, 22 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Analysis of GRPO and its variants
  5. A Discriminative Constrained Optimization Framework
  6. A basic approach
  7. An improved approach for tackling imbalanced rollouts
  8. Experiments
  9. Comparison with Baselines
  10. Ablation Studies
  11. Conclusion
  12. More Experimental Results
  13. Detailed comparison between DisCO and DisCO-b
  14. Training dynamics for fine-tuning 8B model.
  15. Experiments on DAPO-Math-17K dataset.
  16. ...and 4 more sections

Key Result

Proposition 1

Let us consider the objective of GRPO and its variants with the following form: Assume that $f(x, y)$ is non-decreasing function of $x$ such that $f(x, y)=\mathbb I(y>0)y f^+(x, 1) - \mathbb I(y\leq 0) |y|f^-(x, 1)$, where both $f^+, f^-$ are non-decreasing functions of $x$, then we have where $s_{\theta}^+(o, q) = \frac{1}{|o|}\sum_{t=1}^{|o|}f^+\left(\frac{\pi_\theta(o_t|q,o_{<t})}{\pi_{\text

Figures (6)

  • Figure 1: (a) Weight on questions based on correctness probability $p$; (b) Histogram of per-question accuracy evaluated in the GRPO learning; (c) Comparison of the ratio of questions with 100% correctness probability; (d) Comparison of the ratio of questions with 0% correctness probability.
  • Figure 2: Training dynamics of different methods: left two are for fine-tuning DeepSeek-R1-Distill-Qwen-1.5B model and right two are for fine-tuning DeepSeek-R1-Distill-Qwen-7B model. (a), (c) plot the training reward (averaged over generated outputs for questions used in each step) vs the number of training steps (cf. Algorithm \ref{['alg:disco']}); (b), (d) plot the generation entropy vs training steps.
  • Figure 3: Ablation studies: left for comparing DisCO vs DisCO-b; middle and right for comparing clipping with non-clipping scoring functions.
  • Figure 4: Ablation studies: left for comparing KL regularization vs constrained optimization; middle for sensitivity of DisCO w.r.t. the hyperparameter $\tau$; right for contribution of each component.
  • Figure 5: Comparison between DisCO-b and DisCO on different models with different score functions.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Proof