Distilling Knowledge from Heterogeneous Architectures for Semantic Segmentation

TL;DR

The paper addresses the challenge of distilling knowledge for semantic segmentation from heterogeneous architectures (e.g., CNNs and Transformers). It introduces HeteroAKD, a framework that transfers knowledge by projecting intermediate representations into an aligned logits space and adds two mechanisms, KMM and KEM, to mix and evaluate cross-architecture knowledge using label guidance. The approach is validated on Cityscapes, Pascal VOC, and ADE20K, showing significant improvements over state-of-the-art KD methods for heterogeneous teacher-student pairs; ablations confirm the necessity of both KMM and KEM and illuminate the impact of hyper-parameters. Overall, HeteroAKD demonstrates that leveraging diverse knowledge from heterogeneous architectures can yield more accurate and robust semantic segmentation models, with practical implications for model compression and cross-architecture knowledge transfer.

Abstract

Current knowledge distillation (KD) methods for semantic segmentation focus on guiding the student to imitate the teacher's knowledge within homogeneous architectures. However, these methods overlook the diverse knowledge contained in architectures with different inductive biases, which is crucial for enabling the student to acquire a more precise and comprehensive understanding of the data during distillation. To this end, we propose for the first time a generic knowledge distillation method for semantic segmentation from a heterogeneous perspective, named HeteroAKD. Due to the substantial disparities between heterogeneous architectures, such as CNN and Transformer, directly transferring cross-architecture knowledge presents significant challenges. To eliminate the influence of architecture-specific information, the intermediate features of both the teacher and student are skillfully projected into an aligned logits space. Furthermore, to utilize diverse knowledge from heterogeneous architectures and deliver customized knowledge required by the student, a teacher-student knowledge mixing mechanism (KMM) and a teacher-student knowledge evaluation mechanism (KEM) are introduced. These mechanisms are performed by assessing the reliability and its discrepancy between heterogeneous teacher-student knowledge. Extensive experiments conducted on three main-stream benchmarks using various teacher-student pairs demonstrate that our HeteroAKD outperforms state-of-the-art KD methods in facilitating distillation between heterogeneous architectures.

Figures (7)

• Figure 1: Comparison of the vanilla KD methods ((a) and (b)) with our HeteroAKD (c).
• Figure 2: Similarity heatmap of intermediate features measured by centered kernel alignment (CKA). We compare features from ResNet-101 (CNN) and Mix Transformer-B4 (Transformer). Best viewed with zoom in.
• Figure 3: Analysis of IoU metrics for class probabilities predicted by CNN-based and Transformer-based architectures. We choose the first pair of teacher-student models for each mode in Table \ref{['tab:cityscape(b)']} for our analysis.
• Figure 4: An overview of the HeteroAKD framework. Here, we take the “CNN$\rightarrow$Transformer” mode as an example.
• Figure 5: T-SNE visualization of learned feature embeddings (i.e., SegFormer-MiT-B4$\rightarrow$DeepLabV3-Res18) on the Cityscapes dataset. We outline some classes with dash circles in their colors for a clearer view.