Adaptive Deep Reasoning: Triggering Deep Thinking When Needed

TL;DR

This work tackles the trade-off between reasoning depth and computational cost in large language models by enabling autonomous switching between short and long chain-of-thought (CoT) reasoning. It first equips a base model with both reasoning modes through supervised fine-tuning on four data categories, then applies reinforcement learning with a long-short adaptive group-wise reward and a logit-based mode-switching loss to balance the two modes per problem. A hybrid reward model—combining a 7B LLM-based correctness signal with formatting checks—along with adaptive reward shaping and a soft length penalty, guides the model to choose the appropriate CoT length based on problem complexity, quantified via accuracy signals. Experiments on diverse mathematical benchmarks show the model can dynamically switch CoT modes with minimal loss in final accuracy while reducing response length, improving practicality for real-world deployments of reasoning-heavy LLMs.

Abstract

Large language models (LLMs) have shown impressive capabilities in handling complex tasks through long-chain reasoning. However, the extensive reasoning steps involved can significantly increase computational costs, posing challenges for real-world deployment. Recent efforts have focused on optimizing reasoning efficiency by shortening the Chain-of-Thought (CoT) reasoning processes through various approaches, such as length-aware prompt engineering, supervised fine-tuning on CoT data with variable lengths, and reinforcement learning with length penalties. Although these methods effectively reduce reasoning length, they still necessitate an initial reasoning phase. More recent approaches have attempted to integrate long-chain and short-chain reasoning abilities into a single model, yet they still rely on manual control to toggle between short and long CoT. In this work, we propose a novel approach that autonomously switches between short and long reasoning chains based on problem complexity. Our method begins with supervised fine-tuning of the base model to equip both long-chain and short-chain reasoning abilities. We then employ reinforcement learning to further balance short and long CoT generation while maintaining accuracy through two key strategies: first, integrating reinforcement learning with a long-short adaptive group-wise reward strategy to assess prompt complexity and provide corresponding rewards; second, implementing a logit-based reasoning mode switching loss to optimize the model's initial token choice, thereby guiding the selection of the reasoning type. Evaluations on mathematical datasets demonstrate that our model can dynamically switch between long-chain and short-chain reasoning modes without substantially sacrificing performance. This advancement enhances the practicality of reasoning in large language models for real-world applications.