Mitigating Overthinking in Large Reasoning Models via Difficulty-aware Reinforcement Learning

TL;DR

This work tackles overthinking in large reasoning models by introducing DiPO, a difficulty-aware reinforcement learning framework that teaches LRMs to infer task complexity and adapt reasoning depth accordingly. It combines a self-reasoning-based difficulty signal with a difficulty-enhanced reward and a GRPO-based policy update to aggressively compress thought for easy tasks while preserving reasoning quality for hard tasks. The approach yields substantial inference efficiency gains with minimal or no loss in accuracy across diverse in-domain and out-of-domain benchmarks, and shows strong cross-domain generalization. These results highlight the value of explicit difficulty modeling for resource-efficient reasoning in LRMs and suggest broad applicability to real-world, latency-sensitive applications.

Abstract

Large Reasoning Models (LRMs) achieve explicit chain-of-thought expansion by imitating deep thinking behaviors of humans, demonstrating excellent performance in complex task scenarios. However, the deep-thinking mode often leads to unnecessarily lengthy reasoning and resource inefficiency when handling simple tasks. This overthinking phenomenon may arise from the generation preference triggered by the reward function during post-training. Existing research attempts to mitigate overthinking from the perspective of prompt design or model training, but generally underestimates the importance of task difficulty awareness, which makes it difficult for LRMs to effectively allocate reasoning resources. In this paper, we propose Difficulty-aware Policy Optimization (DiPO), a reinforcement learning-based LRM training framework. DiPO encourages LRM to spontaneously model task complexity, and integrates them into reinforcement learning framework to adjust the generation preferences introduced by post-training. A difficulty modeling method based on model self-reasoning is proposed, which significantly reduces the dependence on manual annotation and formalize task complexity. We further develop a difficulty-signal-enhanced reward function that incorporates a penalty for lengthy reasoning while considering reasoning performance and output format. Experimental results indicate that DiPO enables the model to spontaneously adjust inference overhead, significantly reducing redundant tokens without losing performance due to thought compression.

Figures (9)

• Figure 1: Responses from different models to the same simple question. This example was tested in June 2025.
• Figure 2: Response Length Reduction Across Different Datasets. Response length reduction across different datasets, with the y-axis representing percentage token reduction for math reasoning (left) and OOD datasets (right).
• Figure 3: Performance of DiPO with Different Token Budgets. The pass rate (y-axis) across various benchmarks is shown as a function of the number of tokens used (x-axis).
• Figure 4: Multi-model performance comparison across difficulty levels on the Math-500 dataset. The x-axis represents the difficulty level of the tasks, and the bars correspond to the response lengths of the Qwen3-4B model (Base) and the DiPO, with accuracy shown by the lines for both models.
• Figure 5: Analysis of the Proportion of Thinking Cost. Average token distribution comparison between the thinking tokens (within the <think> tag) and solution tokens across different datasets.