Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Large Reasoning Models Effective Reflection

Hanbin Wang, Jingwei Song, Jinpeng Li, Qi Zhu, Fei Mi, Ganqu Cui, Yasheng Wang, Lifeng Shang

TL;DR

This work tackles superficial reflection in Large Reasoning Models by introducing a two-stage framework: Self-Critique Fine-Tuning (SCFT) to bootstrap reflection through self-generated critiques, and Reinforcement Learning with Effective Reflection Rewards (RLERR) to internalize self-correction via hierarchical, reflection-aware rewards. It defines Effective Reflection Ratio (ERR) to quantify reflection quality and demonstrates substantial gains in reasoning accuracy on AIME2024/2025 benchmarks, with additional improvements in reflection behavior. The approach leverages self-generated data, rejection sampling for data quality, and a reward model to guide dense feedback, achieving state-of-the-art results among similarly sized models. The work highlights the importance of initializing RL with strong reflective priors and combining outcome-based and reflection-based rewards for robust, non-superficial self-correction, with broader implications for scalable, reflective reasoning in LRMs.

Abstract

Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.

Teaching Large Reasoning Models Effective Reflection

TL;DR

This work tackles superficial reflection in Large Reasoning Models by introducing a two-stage framework: Self-Critique Fine-Tuning (SCFT) to bootstrap reflection through self-generated critiques, and Reinforcement Learning with Effective Reflection Rewards (RLERR) to internalize self-correction via hierarchical, reflection-aware rewards. It defines Effective Reflection Ratio (ERR) to quantify reflection quality and demonstrates substantial gains in reasoning accuracy on AIME2024/2025 benchmarks, with additional improvements in reflection behavior. The approach leverages self-generated data, rejection sampling for data quality, and a reward model to guide dense feedback, achieving state-of-the-art results among similarly sized models. The work highlights the importance of initializing RL with strong reflective priors and combining outcome-based and reflection-based rewards for robust, non-superficial self-correction, with broader implications for scalable, reflective reasoning in LRMs.

Abstract

Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.
Paper Structure (25 sections, 7 equations, 5 figures, 6 tables)

This paper contains 25 sections, 7 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Behaviors of LRMs
  4. Critique Learning
  5. Reinforcement Learning for Reasoning
  6. Methodology
  7. Effective Reflection
  8. Self-Critique Fine-Tuning (SCFT)
  9. Construction of Training Data
  10. Training Objective
  11. Reinforcement Learning with Effective Reflection Rewards (RLERR)
  12. Hierarchical Reflection Principles
  13. Training Setup
  14. Experimental Methodology
  15. Evaluation Results
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Total number of reflections in correct and incorrect responses on AIME2024. The blue and grey bars respectively show the total occurrences of reflection keywords in correct and incorrect responses. The Effective Reflection Ratio (ERR) is the proportion of reflection keywords in the correct responses, which is defined in Section \ref{['model:def']}. DS represents DeepSeek. Overall, the model is facing serious problems of ineffective reflection.
  • Figure 2: The overall pipeline of our proposed SCFT and RLERR method.
  • Figure 3: The impact of the amount of SCFT and Self-Distill training data on the model performance. The base model is DeepSeek-R1-Distill-Qwen-1.5B. The Pass@1 represents the average performance across all test sets.
  • Figure 4: Analysis of DeepScaleR-1.5B-Preview RL Dynamics on AIME2024. (a) The SCFT-initialized model (green) achieves a higher performance ceiling. (b) RLERR (combining outcome and effective reflection rewards) significantly outperforms using outcome or reflection rewards in isolation.
  • Figure 5: The Pass@16 results of different models. "Base" represents the model that has not been fine-tuned, and "SD" and "SC" respectively represent the models fine-tuned through Self-Distill and Self-Critique Fine-tuning.