Towards Flash Thinking via Decoupled Advantage Policy Optimization

TL;DR

The paper tackles overthinking in large reasoning models by introducing DEPO, a reinforcement learning framework with three innovations: decoupled advantage computation at the token level to suppress inefficient reasoning, a difficulty-aware length penalty to shorten responses adaptively, and an advantage clipping mechanism to stabilize policy updates. A Generative Reward Model (GRM) identifies the first correct reasoning step and partitions outputs into efficient and inefficient segments, enabling token-level learning signals. Empirical results on DeepSeek-Distill-Qwen-7B and -1.5B show DEPO substantially reduces response length (≈38–39%) with maintained or improved accuracy, and ablations confirm the superiority of decoupled advantages for suppressing redundant thinking. The approach advances efficient reasoning for math problem solving and offers a principled pathway to faster, cost-effective LRMs without sacrificing performance.

Abstract

Recent Large Reasoning Models (LRMs) have achieved remarkable performance in solving complex problems via supervised fine-tuning (SFT) and reinforcement learning (RL). Although existing RL algorithms significantly enhance model accuracy, they still suffer from excessively lengthy responses and overthinking issues, resulting in increased inference latency and computational consumption, especially for simple tasks that require minimal reasoning. To address this, we propose a novel RL framework, DEPO, to reduce inefficient reasoning for models. Our method mainly consists of three core components: (1) an innovative advantage decoupled algorithm to guide model reduction of inefficient tokens; (2) a difficulty-aware length penalty to lower the overall length of model responses; (3) an advantage clipping method to prevent bias in policy optimization. In our experiments, applied to DeepSeek-Distill-Qwen-7B and DeepSeek-Distill-Qwen-1.5B as base models, DEPO achieves a significant reduction in sequence length by 39% and reduces excessive reasoning paths in inefficient tokens, while outperforming the base model in overall accuracy.

Figures (8)

• Figure 1: Illustration of proposed DEPO. DEPO enables token-level advantage estimation to update $\pi_{old}$ to $\pi_{\theta}$, in contrast to sequence-level methods in GRPO.
• Figure 2: Scoring accuracy of rule and GRM across difficulty levels.
• Figure 3: The redundant reasoning matching method of DEPO
• Figure 4: Training accuracy comparison of naive GRPO, length penalty with adv_clip and length penalty without adv_clip.
• Figure 5: Comparison of model accuracy and response length on DeepScaleR between naive GRPO and DEPO at different training steps.