Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs

Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang

TL;DR

PLaD introduces a preference-based distillation framework for large language models that circumvents API-only teacher access by generating pseudo-preference pairs from teacher–student capacity gaps and optimizing a calibration loss to align the student’s output likelihood with relative quality. By leveraging ranking and margin-based losses and a score function, PLaD steers the student toward producing outputs that are not only accurate but also well-calibrated in quality. Across TL;DR and Anthropic-HH tasks and multiple model families (e.g., LLaMA-2, GPT-Neo, PaLM-2, and T5), PLaD achieves superior win-rate improvements and ROUGE scores compared to standard KD methods, and shows robustness to generation length and data scales. The approach is annotation-free, scalable, and applicable to diverse LLMs, offering a practical path to efficient and effective LLM distillation in real-world settings.

Abstract

Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. However, traditional KD techniques face specific challenges when applied to LLMs, including restricted access to LLM outputs, significant teacher-student capacity gaps, and the inherited mis-calibration issue. In this work, we present PLaD, a novel preference-based LLM distillation framework. PLaD exploits the teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. Then, PLaD leverages a ranking loss to re-calibrate student's estimation of sequence likelihood, which steers the student's focus towards understanding the relative quality of outputs instead of simply imitating the teacher. PLaD bypasses the need for access to teacher LLM's internal states, tackles the student's expressivity limitations, and mitigates the student mis-calibration issue. Through extensive experiments on two sequence generation tasks and with various LLMs, we demonstrate the effectiveness of our proposed PLaD framework.

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs

TL;DR

PLaD introduces a preference-based distillation framework for large language models that circumvents API-only teacher access by generating pseudo-preference pairs from teacher–student capacity gaps and optimizing a calibration loss to align the student’s output likelihood with relative quality. By leveraging ranking and margin-based losses and a score function, PLaD steers the student toward producing outputs that are not only accurate but also well-calibrated in quality. Across TL;DR and Anthropic-HH tasks and multiple model families (e.g., LLaMA-2, GPT-Neo, PaLM-2, and T5), PLaD achieves superior win-rate improvements and ROUGE scores compared to standard KD methods, and shows robustness to generation length and data scales. The approach is annotation-free, scalable, and applicable to diverse LLMs, offering a practical path to efficient and effective LLM distillation in real-world settings.

Abstract

Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. However, traditional KD techniques face specific challenges when applied to LLMs, including restricted access to LLM outputs, significant teacher-student capacity gaps, and the inherited mis-calibration issue. In this work, we present PLaD, a novel preference-based LLM distillation framework. PLaD exploits the teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. Then, PLaD leverages a ranking loss to re-calibrate student's estimation of sequence likelihood, which steers the student's focus towards understanding the relative quality of outputs instead of simply imitating the teacher. PLaD bypasses the need for access to teacher LLM's internal states, tackles the student's expressivity limitations, and mitigates the student mis-calibration issue. Through extensive experiments on two sequence generation tasks and with various LLMs, we demonstrate the effectiveness of our proposed PLaD framework.
Paper Structure (28 sections, 8 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 28 sections, 8 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Auto-Regressive Text Generation
  5. Sequence Knowledge Distillation
  6. PLaD: Preference-based Large Language Model Distillation
  7. Framework Overview
  8. Pseudo Preference Pairs Generation
  9. Distillation with Preference Pairs
  10. Experiments
  11. Experiment Setup
  12. Main Results
  13. Impact of Real Preference Pairs
  14. Performance in Length Ranges
  15. Scaling with Distillation Data
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Our PLaD framework starts with separate SFT processes for both the teacher and student models. The best checkpoint of the teacher model is selected based on the win rate over targets, while the student model went through the whole SFT for initialization. In the next stage, we generate pseudo-preference data by sampling generation pairs from the teacher and the student. The student model then undergoes preference distillation using this pseudo-preference data to produce a distilled student model.
  • Figure 2: Comparison between using real preference pairs compared to using pseudo preference pairs.
  • Figure 3: The $\triangle$ win rate against the initial student v.s. the length of student model generation. Experiments are conducted on TL-DR with LLaMA-2.
  • Figure 4: Scaling properties of the distillation data.
  • Figure 5: The case study on the TL;DR dataset.