Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning

Xiaoxue Cheng, Junyi Li, Zhenduo Zhang, Xinyu Tang, Wayne Xin Zhao, Xinyu Kong, Zhiqiang Zhang

TL;DR

LRMs often overthink, producing redundant reasoning paths. ACPO tackles this by enabling dynamic switching between fast and slow thinking through system-aware reasoning tokens and an online token length budget, integrated into a GRPO-based reinforcement learning framework with a two-stage training (SFT cold start followed by RL). The approach achieves substantial reductions in reasoning length while maintaining competitive accuracy across GSM8K, MATH 500, and AIME 2024, demonstrating difficulty-aware and interpretable cognitive allocation. While promising, ACPO may face challenges in open-domain generalization and can incur accuracy-efficiency trade-offs, suggesting avenues for more generalizable difficulty estimation and reward design.

Abstract

Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch. ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.

Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning

TL;DR

LRMs often overthink, producing redundant reasoning paths. ACPO tackles this by enabling dynamic switching between fast and slow thinking through system-aware reasoning tokens and an online token length budget, integrated into a GRPO-based reinforcement learning framework with a two-stage training (SFT cold start followed by RL). The approach achieves substantial reductions in reasoning length while maintaining competitive accuracy across GSM8K, MATH 500, and AIME 2024, demonstrating difficulty-aware and interpretable cognitive allocation. While promising, ACPO may face challenges in open-domain generalization and can incur accuracy-efficiency trade-offs, suggesting avenues for more generalizable difficulty estimation and reward design.

Abstract

Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch. ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.

Paper Structure

This paper contains 33 sections, 8 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Group Relative Policy Optimization (GRPO)
  4. Token Length Budget
  5. Method
  6. Explicit Dual Process Reasoning
  7. Data Construction
  8. Supervised Fine-Tuning (Cold Start)
  9. Reinforcement Learning with ACPO
  10. Reward Design
  11. Accuracy Reward.
  12. Online TLB Reward.
  13. System Pattern Reward.
  14. Experiment
  15. Experimental Setup
  16. ...and 18 more sections

Figures (4)

  • Figure 1: The overall framework of ACPO. The upper section illustrates the system explicitization process and cold start training via SFT. The lower section presents the ACPO training phase.
  • Figure 2: Average response length and accuracy across different difficulty levels on MATH 500.
  • Figure 3: Average fast and slow thinking ratios across different difficulty levels on MATH 500.
  • Figure 4: An case study comparing the reasoning process of DeepSeek-R1-Distill-Qwen-1.5B trained with GRPO and ACPO in MATH 500 Dataset.