Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning While Staying Curious: Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

Hao Wang, Hao Gu, Hongming Piao, Kaixiong Gong, Yuxiao Ye, Xiangyu Yue, Sirui Han, Yike Guo, Dapeng Wu

TL;DR

The paper tackles entropy collapse during SFT in the SFT-then-RL paradigm, which can hinder exploration in the subsequent RL stage. It introduces CurioSFT, a two-component method comprising Self-Exploratory Distillation (SED) and Entropy-Guided Temperature Selection (EGTS) to preserve useful entropy while avoiding knowledge forgetting. Empirical results on mathematical reasoning benchmarks show that CurioSFT improves both in-distribution and out-of-distribution performance during SFT and yields substantial RL-stage gains when paired with GRPO, surpassing vanilla SFT and several baselines. The work demonstrates robustness across model backbones and provides a practical approach to unlock higher RL performance ceilings by maintaining curiosity-driven exploration during SFT. Overall, CurioSFT offers a principled, token-aware mechanism to balance exploration and knowledge retention, with potential broad impact on reasoning-enabled LLM pipelines in verification-heavy tasks.

Abstract

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.

Learning While Staying Curious: Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

TL;DR

The paper tackles entropy collapse during SFT in the SFT-then-RL paradigm, which can hinder exploration in the subsequent RL stage. It introduces CurioSFT, a two-component method comprising Self-Exploratory Distillation (SED) and Entropy-Guided Temperature Selection (EGTS) to preserve useful entropy while avoiding knowledge forgetting. Empirical results on mathematical reasoning benchmarks show that CurioSFT improves both in-distribution and out-of-distribution performance during SFT and yields substantial RL-stage gains when paired with GRPO, surpassing vanilla SFT and several baselines. The work demonstrates robustness across model backbones and provides a practical approach to unlock higher RL performance ceilings by maintaining curiosity-driven exploration during SFT. Overall, CurioSFT offers a principled, token-aware mechanism to balance exploration and knowledge retention, with potential broad impact on reasoning-enabled LLM pipelines in verification-heavy tasks.

Abstract

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.
Paper Structure (35 sections, 2 theorems, 20 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 35 sections, 2 theorems, 20 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. SFT Loss.
  4. Entropy Loss.
  5. The Pitfall of Entropy Loss in SFT
  6. Ungrounded entropy degrades exploration capability.
  7. Token-agnostic regularization amplifies knowledge forgetting.
  8. Proposed Solution: CurioSFT
  9. Self-Exploratory Distillation
  10. Entropy-Guided Temperature Selection
  11. Experiments
  12. Training.
  13. Evaluation.
  14. Effectiveness of CurioSFT
  15. Baselines.
  16. ...and 20 more sections

Key Result

Theorem 1

Let $\pi_\theta(\cdot \mid \mathbf{s};\tau)=\mathrm{softmax}\!(z_\theta(\cdot\mid\mathbf{s})/\tau)$ be the distribution derived from logits $z_\theta(\cdot\mid\mathbf{s})$ with temperature $\tau$. Then the entropy $H\!(\pi_\theta(\cdot \mid \mathbf{s};\tau))$ is non-decreasing with respect to $\tau$

Figures (5)

  • Figure 1: Evaluation entropy and accuracy (Avg@8 in AIME 2024) across the SFT and RL stages. CurioSFT mitigates entropy collapse during SFT, and yields larger accuracy gains in the RL stage.
  • Figure 2: Expert token entropy and evaluation accuracy during the SFT stage. Compared to vanilla entropy loss, which uniformly encourages entropy across tokens, CurioSFT selectively increases entropy on high-entropy tokens while preserving low-entropy ones. We further observe that the increased token entropy induced by entropy loss does not translate into actual improvements in Pass@32 performance in our experiments.
  • Figure 3: Proposed solution: CurioSFT
  • Figure 4: Accuracy vs. N-gram diversity.
  • Figure 5: Sensitivity to the entropy pivot $H_{\text{pivot}}$.

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Theorem 2: Temperature scaling is the optimum of P1
  • proof