Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning

TL;DR

This work addresses reward hacking in reinforcement-learning-based hybrid reasoning models that toggle between thinking (CoT) and non-thinking modes. The authors introduce Thinking-Based Non-Thinking (TNT), which derives a per-query non-thinking token cap from the thinking-mode solution component, avoiding the computational burden of SFT and improving detection of reward-hacking signals. TNT integrates this adaptive cap into a GRPO-based RL objective with a carefully designed reward function, achieving lower token usage while improving accuracy on five mathematical benchmarks, and showing strong generalization to larger base models and out-of-distribution tasks. The approach demonstrates that adaptive, solution-aware resource budgeting can significantly enhance efficiency and reliability in hybrid reasoning systems, offering a practical path forward for scalable reasoning with LRMs.

Abstract

Large reasoning models (LRMs) have attracted much attention due to their exceptional performance. However, their performance mainly stems from thinking, a long Chain of Thought (CoT), which significantly increase computational overhead. To address this overthinking problem, existing work focuses on using reinforcement learning (RL) to train hybrid reasoning models that automatically decide whether to engage in thinking or not based on the complexity of the query. Unfortunately, using RL will suffer the the reward hacking problem, e.g., the model engages in thinking but is judged as not doing so, resulting in incorrect rewards. To mitigate this problem, existing works either employ supervised fine-tuning (SFT), which incurs high computational costs, or enforce uniform token limits on non-thinking responses, which yields limited mitigation of the problem. In this paper, we propose Thinking-Based Non-Thinking (TNT). It does not employ SFT, and sets different maximum token usage for responses not using thinking across various queries by leveraging information from the solution component of the responses using thinking. Experiments on five mathematical benchmarks demonstrate that TNT reduces token usage by around 50% compared to DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B, while significantly improving accuracy. In fact, TNT achieves the optimal trade-off between accuracy and efficiency among all tested methods. Additionally, the probability of reward hacking problem in TNT's responses, which are classified as not using thinking, remains below 10% across all tested datasets.

Figures (8)

• Figure 1: Overview of TNT.
• Figure 2: Average accuracy and token usage comparison across different hybrid reasoning model training methods on mathematical benchmarks. We only presented the evaluation results of their open-source checkpoints while some of these methods lack the trained checkpoints based on DeepScaleR-1.5B, and DeepSeek-R1-Distill-Qwen-7B. Adapt-$x$ refers to AdaptThink with $\delta=x*0.01$. Auto-S$x$ refers to AutoThink-Stage $x$. We also follow these abbreviation conventions in the table below.
• Figure 3: An example of the reward hacking problem occurs when Assistant uses the AutoThink model tu2025learning. The first generated token in the response is $\text{</think>}$ ($\text{<think>}$ is included in the input prompt $x$), indicating that the response is classified as the non-thinking mode. However, the response clearly demonstrates characteristics of the thinking mode, as evidenced by the usage of keywords like "Wait" and "Alternatively" along with the regeneration of the thinking's termination token $\text{</think>}$. The answer $204$ is correct. This response is undeniably part of the thinking mode despite being incorrectly assigned a higher reward associated with the correct answer of the non-thinking mode. This discrepancy results in a clear instance of the reward hacking problem, where the reward allocation does not align with the true mode of the response.
• Figure 4: Probability of thinking-related verbs appearing in non-thinking mode responses across models with base model DeepSeek-R1-Distill-Qwen-1.5B on mathematical benchmarks.
• Figure 5: Accuracy (left), token usage (middle), and non-thinking ratio (right) in RL training on DeepSeek-R1-Distill-Qwen-1.5B. Due to constraints on computational resources, we only use AIME24 as the test dataset.