Knowledge Distillation Meets Open-Set Semi-Supervised Learning

TL;DR

This work tackles distilling structured, semantically rich representations by introducing Semantic Representational Distillation (SRD), which uses the pretrained teacher's classifier as a semantic critic via a cross-network logit $\hat{\bm{z}} = h^{t}(\varphi(\bm{x}^{s}))$. It further extends SRD to open-set semi-supervised learning by leveraging unconstrained unlabeled data and treating seen classes as a semantic basis to approximate unseen classes, enabling effective knowledge transfer beyond closed-set assumptions. Empirically, SRD outperforms state-of-the-art knowledge distillation and open-set SSL methods across CIFAR-10/100, ImageNet-1K, face recognition, and binary-network distillation, while revealing limitations of OOD-based SSL approaches. The work demonstrates that leveraging a teacher as semantic guidance can yield robust improvements and provides a practical, scalable approach with code available at the authors’ repository.

Abstract

Existing knowledge distillation methods mostly focus on distillation of teacher's prediction and intermediate activation. However, the structured representation, which arguably is one of the most critical ingredients of deep models, is largely overlooked. In this work, we propose a novel {\em \modelname{}} ({\bf\em \shortname{})} method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. The key idea is that we leverage the teacher's classifier as a semantic critic for evaluating the representations of both teacher and student and distilling the semantic knowledge with high-order structured information over all feature dimensions. This is accomplished by introducing a notion of cross-network logit computed through passing student's representation into teacher's classifier. Further, considering the set of seen classes as a basis for the semantic space in a combinatorial perspective, we scale \shortname{} to unseen classes for enabling effective exploitation of largely available, arbitrary unlabeled training data. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL). Extensive experiments show that our \shortname{} outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks, as well as less studied yet practically crucial binary network distillation. Under more realistic open-set SSL settings we introduce, we reveal that knowledge distillation is generally more effective than existing Out-Of-Distribution (OOD) sample detection, and our proposed \shortname{} is superior over both previous distillation and SSL competitors. The source code is available at \url{https://github.com/jingyang2017/SRD\_ossl}.

Figures (8)

• Figure 1: Schematic overview of the proposed Semantic Representational Distillation (SRD) method for knowledge distillation at the presence of both labeled and unlabeled data. Following the knowledge distillation pipeline, (a) we first pretrain a teacher model on the labeled training set. (b) Subsequently, we distil the semantic knowledge from the pretrained frozen teacher to improve the optimization of a student. Specifically, given a training image $I$, we feed it into both the teacher $T$ and the student $S$ to obtain the feature representations $\bm{x}^t$ and $\bm{x}^s$. Critically, we introduce a notion of cross-network logit $\hat{\bm{z}}$, obtained by passing the student's representation $\bm{x}^s$ into the teacher's classifier $h^t$ via a feature adaptor $\varphi$. Considering the teacher's classifier $h^t$ as a semantic critic, we distil the semantic knowledge of $\bm{x}^t$ to $\bm{x}^s$ via aligning the cross-network logit $\hat{\bm{z}}$ towards to the teacher's logit $\bm{z}^t$. In this design, the two representations $\bm{x}^t$ and $\bm{x}^s$ share the same semantic critic ( i.e., classifier) which could facilitate representational knowledge distillation. To further ease the semantic distillation, we impose a feature-level alignment regularization $\mathcal{R}$ between the teacher's representation $\bm{x}^t$ and the adapted student's representation $\hat{\bm{x}}$. For labeled training samples, we also apply a supervised learning supervision on the student's prediction.
• Figure 2: Feature distribution visualization of 10 classes on CIFAR-100. Class is color coded. Better viewed in color.
• Figure 3: Classifier confidence distribution over TP, FP, TN, FN by SRD and KD on CIFAR-100 test set.
• Figure 4: Per-epoch usage of unlabeled data (those surviving through OOD detection) with ( top) T2T T2T and ( bottom) OpenMatch (OM) openmatch.
• Figure 5: Size effect of unlabeled data selected ( Top) randomly or ( Bottom) by the teacher prediction score. Teacher: WRN40-2. Student: WRN40-1.