Multi-perspective Contrastive Logit Distillation

TL;DR

This paper revisits knowledge distillation by reframing logit distillation through the semantic lens of raw logits and introduces Multi-perspective Contrastive Logit Distillation (MCLD). MCLD leverages three contrastive perspectives—Instance-wise, Sample-wise, and Category-wise CLD—plus a Target Mask, enabling 1-to-N comparisons without extra feature-mapping modules and using a warm-up schedule to emphasize category-level signals later in training. Empirically, MCLD achieves state-of-the-art performance on CIFAR-100 and ImageNet, often surpassing both logit- and feature-distillation baselines while exhibiting superior training efficiency, and extends effectively to Vision Transformers and transfer tasks. The work highlights the untapped potential of logits for knowledge transfer and provides a practical, scalable framework for robust distillation across diverse architectures and datasets.

Abstract

In previous studies on knowledge distillation, the significance of logit distillation has frequently been overlooked. To revitalize logit distillation, we present a novel perspective by reconsidering its computation based on the semantic properties of logits and exploring how to utilize it more efficiently. Logits often contain a substantial amount of high-level semantic information; however, the conventional approach of employing logits to compute Kullback-Leibler (KL) divergence does not account for their semantic properties. Furthermore, this direct KL divergence computation fails to fully exploit the potential of logits. To address these challenges, we introduce a novel and efficient logit distillation method, Multi-perspective Contrastive Logit Distillation (MCLD), which substantially improves the performance and efficacy of logit distillation. In comparison to existing logit distillation methods and complex feature distillation methods, MCLD attains state-of-the-art performance in image classification, and transfer learning tasks across multiple datasets, including CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10. Additionally, MCLD exhibits superior training efficiency and outstanding performance with distilling on Vision Transformers, further emphasizing its notable advantages. This study unveils the vast potential of logits in knowledge distillation and seeks to offer valuable insights for future research.

Figures (6)

• Figure 1: Training time (per batch) vs. Top-1 Accuracy on CIFAR-100 cifar100. We set ResNet32×4 as the teacher and ResNet8x4 as the student.
• Figure 2: Differences between raw logits, class probabilities (softmaxed logits), and softmaxed logits with temperature ($\tau$) from the pre-trained teacher model (ResNet32x4). The sample was randomly selected from the CIFAR-100 cifar100 dataset.
• Figure 3: Visualization of the penultimate feature map (mean in channel-wise after activation) and raw logits.
• Figure 4: The t-SNE tsne visualization of features in a homogeneous structure. We set ResNet32×4 as the teacher model and ResNet8×4 as the student model. Our method achieves more separate classification results.
• Figure 5: Difference in correlation matrices between the student's and teacher's logits in a heterogeneous structure. We set WRN-40-2 as the teacher and ResNet8x4 as the student. The lighter the shade, the smaller the difference.