Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

Wenjing Zhang, Jiangze Yan, Jieyun Huang, Yi Shen, Shuming Shi, Ping Chen, Ning Wang, Zhaoxiang Liu, Kai Wang, Shiguo Lian

TL;DR

Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap, is proposed and extensive experiments demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.

Abstract

Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat the teacher as a static filter, discarding complex "corner-case" problems where the teacher fails to explore valid solutions independently, thereby creating an artificial "Teacher Ceiling" for the student. In this work, we propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap. Drawing on the educational theory of the Zone of Proximal Development(ZPD), HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to frontier breakthrough. Extensive experiments on multiple benchmarks demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.

HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

TL;DR

Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap, is proposed and extensive experiments demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.

Abstract

Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat the teacher as a static filter, discarding complex "corner-case" problems where the teacher fails to explore valid solutions independently, thereby creating an artificial "Teacher Ceiling" for the student. In this work, we propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap. Drawing on the educational theory of the Zone of Proximal Development(ZPD), HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to frontier breakthrough. Extensive experiments on multiple benchmarks demonstrate that HEAL significantly outperforms traditional SFT distillation and other baselines.
Paper Structure (25 sections, 7 equations, 5 figures, 2 tables)

This paper contains 25 sections, 7 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Methodology
  5. Synthesis via GEAR
  6. Theoretical Grounding.
  7. Identifying Critical Reasoning Breakpoint.
  8. Filtering via PURE
  9. Training via PACE
  10. Experiments
  11. Experimental Settings
  12. Benchmarks
  13. Teacher Model and Student Models
  14. Baseline Methods
  15. Training Details
  16. ...and 10 more sections

Figures (5)

  • Figure 1: (A) The “Teacher Ceiling” challenge of standard rejection sampling for reasoning distillation. (B) Hindsight prompts with hint help obtain valid reasoning trajectories for difficult problems, thereby closer to the “Teacher Ceiling”.
  • Figure 2: The HEAL framework for reasoning distillation.
  • Figure 3: Providing the ground-truth answer as a global hint in the prompt.
  • Figure 4: Providing the ground-truth answer and partial trajectory as a global hint in the prompt.
  • Figure 5: Sensitivity analysis of $\lambda$ values on the OlympiadBench benchmark.