Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models

TL;DR

This paper tackles the challenge of aligning small language models with human preferences by moving beyond pairwise teacher signals. It introduces Preference-Aligned Distillation (PAD), which encodes teacher preferences as a distribution over all possible preferences, enabling nuanced supervisory signals. PAD derives token-level rewards from intrinsic model likelihoods via inverse reinforcement learning and uses a three-phase process: sampling diverse responses, calibrating rewards with MCQ-based selection probabilities, and distilling the teacher's preference distribution through Vanilla and Probabilistic losses, aided by a Preference Decomposing Strategy to reduce compute. Empirical results on Gemma-2 and LLaMA-3 families across Alpaca Eval 2.0, Arena-Hard, MT-Bench, and GSM8K show that PAD consistently outperforms traditional KD and prior preference distillation methods, with PAD-PPD achieving notable gains and even surpassing teacher performance on MT-Bench in some setups. The work demonstrates a practical and scalable path to aligning SLMs with human values, with broader implications for safe and useful AI systems.

Abstract

Aligning small language models (SLMs) with human values typically involves distilling preference knowledge from large language models (LLMs). However, existing distillation methods model preference knowledge in teacher LLMs by comparing pairwise responses, overlooking the extent of difference between responses. This limitation hinders student SLMs from capturing the nuanced preferences for multiple responses. In this paper, we propose a Preference-Aligned Distillation (PAD) framework, which models teacher's preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals. Our insight in developing PAD is rooted in the demonstration that language models can serve as reward functions, reflecting their intrinsic preferences. Based on this, PAD comprises three key steps: (1) sampling diverse responses using high-temperature; (2) computing rewards for both teacher and student to construct their intrinsic preference; and (3) training the student's intrinsic preference distribution to align with the teacher's. Experiments on four mainstream alignment benchmarks demonstrate that PAD consistently and significantly outperforms existing approaches, achieving over 20\% improvement on AlpacaEval 2 and Arena-Hard, indicating superior alignment with human preferences. Notably, on MT-Bench, using the \textsc{Gemma} model family, the student trained by PAD surpasses its teacher, further validating the effectiveness of our PAD.

Figures (6)

• Figure 1: Comparison of the Teacher-as-Annotator methods and our PAD, where "A $\succ$ B" means the LLM prefers response A over B.
• Figure 2: The overall process of the PAD contains three critical steps. The initial step involves sampling diverse responses with high temperature (§\ref{['sec:phase-1']}). Next, rewards for both models are computed, where the rewards of the teacher would be calibrated(§\ref{['sec:phase-2']}). Finally, the student is trained to mimic the teacher's preference distributions.(§\ref{['sec:phase-3']})
• Figure 3: Iterative Distillation Process.
• Figure 4: Alpaca-Eval LC with different iterations.
• Figure 5: Alpaca-Eval LC Win Rate with different $\alpha$