Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?

TL;DR

The paper tackles knowledge distillation by transferring both data and reward signals from a teacher LLM to a smaller student, addressing the unreliability of teacher-based evaluations. It introduces a three-stage pipeline—data distillation with multi-temperature teacher outputs, SFT warm-up, and self-supervised reward distillation that learns a reward model to guide PPO-based RL. Empirical results on GSM8K and MMLU-PRO show the approach consistently outperforms traditional SFT baselines and, in several configurations, lets the student exceed the teacher. This scalable strategy reduces dependence on external reward supervision while preserving performance gains across tasks and model sizes.

Abstract

Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT). However, this approach neglects the potential to distill both data (output content) and reward signals (quality evaluations). Extracting reliable reward signals directly from teacher models is challenging, as LLMs are optimized for generation rather than evaluation, often resulting in biased or inconsistent assessments. To address this limitation, we propose a novel distillation pipeline that transfers both responses and rewards. Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses, enabling reward learning without explicit external evaluation. The reward model subsequently guides reinforcement learning (RL), allowing iterative refinement of the student model after an SFT warm-up phase. Experiments on GSM8K and MMLU-PRO demonstrate that our method consistently outperforms traditional SFT-based approaches, enabling student models to surpass the performance of their teachers. This work highlights the potential for scalable, efficient distillation through structured self-supervised reward learning, reducing dependence on external reward supervision.

Figures (9)

• Figure 1: Comparison of (a) traditional knowledge distillation approaches using supervised fine-tuning (SFT) to distill data directly into a student model, and (b) our proposed method, which distills both data (output content) and rewards (quality evaluations) in a self-supervised manner. By training a reward model (RM) and applying reinforcement learning (RL), the student model progressively surpasses the teacher's performance.
• Figure 2: The overall framework of the proposed method. Data Distillation: Teacher LLMs first generate and self-evaluate their answers, then select high-quality responses for the student model warm-up via supervised fine-tuning (SFT). Reward Distillation: The student is further refined through reinforcement learning using a self-supervised reward mechanism.
• Figure 3: Visualization of the ratio of questions without reliable pseudo labels based on the teacher's evaluation.
• Figure 4: Visualization on teacher's evaluation (pseudo labels) across different datasets and teachers. For each subfigure, we represent the samples which are regarded as incorrect by teacher in the left and correct ones in the right. We represent the judgement by ground truth correct in yellow and incorrect in green.
• Figure 5: Ablation on $\lambda$ in reward model learning. We keep the evaluation set the same.