CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective

TL;DR

This work tackles the limitations of traditional knowledge distillation and contrastive KD by proposing CKD, a sample-wise contrastive framework that enforces intra-sample logit alignment and inter-sample semantic separation using a fixed teacher. It derives a concise, InfoNCE-like loss with positive pairs $(oldsymbol{t}_i,oldsymbol{s}_i)$ and negatives $(oldsymbol{s}_i,oldsymbol{s}_j)$, avoiding reliance on large batches or temperature settings. Across CIFAR-100, ImageNet-1K, Places365, and MS COCO, CKD demonstrates consistent improvements over vanilla KD and competitive results against state-of-the-art methods, with notable gains in heterogeneous teacher-student settings. The approach also shows favorable training efficiency and robust performance when combined with feature-based distillation, suggesting practical applicability to large-scale, cross-architecture knowledge transfer.

Abstract

In this paper, we propose a simple yet effective contrastive knowledge distillation framework that achieves sample-wise logit alignment while preserving semantic consistency. Conventional knowledge distillation approaches exhibit over-reliance on feature similarity per sample, which risks overfitting, and contrastive approaches focus on inter-class discrimination at the expense of intra-sample semantic relationships. Our approach transfers "dark knowledge" through teacher-student contrastive alignment at the sample level. Specifically, our method first enforces intra-sample alignment by directly minimizing teacher-student logit discrepancies within individual samples. Then, we utilize inter-sample contrasts to preserve semantic dissimilarities across samples. By redefining positive pairs as aligned teacher-student logits from identical samples and negative pairs as cross-sample logit combinations, we reformulate these dual constraints into an InfoNCE loss framework, reducing computational complexity lower than sample squares while eliminating dependencies on temperature parameters and large batch sizes. We conduct comprehensive experiments across three benchmark datasets, including the CIFAR-100, ImageNet-1K, and MS COCO datasets, and experimental results clearly confirm the effectiveness of the proposed method on image classification, object detection, and instance segmentation tasks.

Figures (10)

• Figure 1: Comparison of three knowledge distillation approaches. Classic KD aligns feature similarities between paired teacher-student samples, while CRD leverages contrastive learning to align class-wise semantics. In contrast, the proposed CKD preserves sample-wise similarity for intra-sample alignment while capturing structural semantics for inter-sample contrast.
• Figure 2: The overall architecture of the proposed Contrastive Knowledge Distillation framework. The framework incorporates newly designed triplets to optimize intra-sample feature similarity and cross-sample semantics simultaneously. For each input instance, the teacher and student logits form positive pairs, while other samples within the mini-batch serve as negative pairs. This formulation significantly enhances training efficiency through computationally efficient negative sample reuse, effectively addressing the memory constraints typically associated with contrastive learning approaches.
• Figure 3: Visualization of the intra-sample alignment. For the sake of simplicity, we assume that the dimension of logits is 3. The spherical surface represents $\mathbf{s}_i$ satisfying the condition $\epsilon_i=\mathbf{t}_i-\mathbf{s}_i$ for the fixed $\epsilon_i$.
• Figure 4: Visualization of two categories of triples. (a) depicts $\left(\boldsymbol{s}_i,\boldsymbol{t}_i,\boldsymbol{s}_j\right)$, while (b) and (c) show $\left(\boldsymbol{s}_i,\boldsymbol{t}_i,\boldsymbol{t}_j\right)$. The blue line indicates that $\boldsymbol{s}_i$ approaches $\boldsymbol{t}_i$, and the red line represents that $\boldsymbol{s}_i$ is far from $\boldsymbol{s}_j$ or $\boldsymbol{t}_j$.
• Figure 5: Pseudo code of CKD in a Numpy-like style.

Theorems & Definitions (6)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof