Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

Keqin Peng, Yuanxin Ouyang, Xuebo Liu, Zhiliang Tian, Ruijian Han, Yancheng Yuan, Liang Ding

TL;DR

This work addresses the inefficiency of reasoning in RLVR by identifying two structural failures in group-relative optimization: Dilution of Length Baseline and Difficulty-Penalty Mismatch. It proposes Dynamic Decoupled Conditional Advantage (DDCA), which decouples correctness and efficiency by (i) computing length advantages only within the correct-response cluster and (ii) dynamically scaling the penalty based on the group pass rate $\rho = \frac{n}{G}$. DDCA uses a conditional sigmoid length reward and a dynamic RLOO estimator to produce a length advantage that adapts to problem difficulty, preserving deep reasoning on hard tasks while trimming redundancy on easy ones. Empirical results across GSM8K, MATH500, AMC23, and AIME25 show substantial token reductions (up to ~60% on simple tasks and >20% on hard ones) with maintained or improved accuracy, validated on two reasoning backbones and multiple baselines. The work provides a principled, implementable baseline for efficient, high-quality probabilistic reasoning in RLVR systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.

Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

TL;DR

This work addresses the inefficiency of reasoning in RLVR by identifying two structural failures in group-relative optimization: Dilution of Length Baseline and Difficulty-Penalty Mismatch. It proposes Dynamic Decoupled Conditional Advantage (DDCA), which decouples correctness and efficiency by (i) computing length advantages only within the correct-response cluster and (ii) dynamically scaling the penalty based on the group pass rate . DDCA uses a conditional sigmoid length reward and a dynamic RLOO estimator to produce a length advantage that adapts to problem difficulty, preserving deep reasoning on hard tasks while trimming redundancy on easy ones. Empirical results across GSM8K, MATH500, AMC23, and AIME25 show substantial token reductions (up to ~60% on simple tasks and >20% on hard ones) with maintained or improved accuracy, validated on two reasoning backbones and multiple baselines. The work provides a principled, implementable baseline for efficient, high-quality probabilistic reasoning in RLVR systems.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.
Paper Structure (39 sections, 15 equations, 4 figures, 2 tables)

This paper contains 39 sections, 15 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement learning with verifiable rewards (RLVR)
  4. Rule-based Reward Modeling with Length Penalty
  5. Revisiting Length Penalty in RLVR
  6. The Dilution of Length Baseline
  7. Difficulty-Penalty Mismatch
  8. High Difficulty ($\rho \to 0$):
  9. Low Difficulty ($\rho \to 1$):
  10. Methodology
  11. Decoupled Advantage Framework
  12. Dynamic Length Reward and Advantage
  13. Conditional Sigmoid Reward.
  14. Dynamic RLOO Estimation.
  15. Experiments
  16. ...and 24 more sections

Figures (4)

  • Figure 1: Performance and efficiency scaling.Top: Pass@1 accuracy. DDCA (red) outperforms Baseline (blue) on both AMC23 and AIME25) tasks. Bottom: Token consumption. DDCA exhibits a plateauing trend, saving up to 2.5k tokens on AIME25 at the 16k budget.
  • Figure 2: Pass@K comparison. Solid lines represent DDCA (Ours), and dashed lines represent the Baseline. DDCA consistently outperforms the baseline across all $K$ values on both AMC23 (Green) and AIME25 (Blue), demonstrating that our dynamic penalty efficiently prunes redundancy without compromising the diversity of valid reasoning paths.
  • Figure 3: Accuracy breakdown on MATH-500 by difficulty. While performance gains are marginal on easy problems (Level 1-2) where baseline dilution is negligible, DDCA achieves a significant +1.2% boost on hard problems (Level 5). This confirms that decoupling the advantage is crucial for robust optimization in low-pass-rate regimes.
  • Figure 4: Effect of $\beta$ on Accuracy and Token Usage.Top: On MATH500, a moderate $\beta=0.3$ boosts accuracy to $86.5\%$ while reducing tokens. Bottom: On AIME25, accuracy drops as $\beta$ increases ($24.9\% \to 23.1\%$), showing high sensitivity. Blue bars: Accuracy; Orange hatched bars: Token count.