Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias

TL;DR

This work tackles the limitation of traditional knowledge distillation that emphasizes matching absolute probabilities, which can lead to exposure bias and underutilize the relational structure among classes. It introduces Group Relative Knowledge Distillation (GRKD), which distills the teacher's relational preferences by learning a group relative loss $L_{GR}$ together with a soft-label loss $L_{ST}$ and a total objective $L_{Total}=\lambda L_{GR}+(1-\lambda)L_{ST}$, where $L_{GR}=-\sum_{(i,j)\in P}\log\sigma\left(\frac{1}{\tau}(\log q_i-\log q_j)\right)$ and $P=\{(i,j)\,|\,s_i>s_j\}$. The method anneals $\lambda$ from 0 to 1 to balance relational and absolute knowledge. Trained on UltraFeedback with ~60k preference samples and evaluated on four benchmarks across two model families, GRKD yields substantial gains over both traditional KD and preference-based baselines, notably >20% on AlpacaEval 2.0 and an average ~9% improvement. This demonstrates that leveraging relational inductive bias in distillation improves fine-grained discrimination and robustness, offering a practical, scalable direction for knowledge transfer.

Abstract

Knowledge distillation typically transfers knowledge from a teacher model to a student model by minimizing differences between their output distributions. However, existing distillation approaches largely focus on mimicking absolute probabilities and neglect the valuable relational inductive biases embedded in the teacher's relative predictions, leading to exposure bias. In this paper, we propose Group Relative Knowledge Distillation (GRKD), a novel framework that distills teacher knowledge by learning the relative ranking among classes, rather than directly fitting the absolute distribution. Specifically, we introduce a group relative loss that encourages the student model to preserve the pairwise preference orderings provided by the teacher's outputs. Extensive experiments on classification benchmarks demonstrate that GRKD achieves superior generalization compared to existing methods, especially in tasks requiring fine-grained class differentiation. Our method provides a new perspective on exploiting teacher knowledge, focusing on relational structure rather than absolute likelihood.