Simplifying Knowledge Transfer in Pretrained Models

TL;DR

The paper tackles the limitation of single-direction knowledge transfer by enabling bidirectional, data-driven interaction among pretrained models during a single training stage. It introduces Bi-KD, where per-sample teacher roles are dynamically assigned based on ground-truth confidence, and both models learn from each other via KL-based distillation complemented by task losses. The framework extends to dense tasks (semantic segmentation, video saliency) and to multiple-model settings, achieving consistent gains across image classification, segmentation, and video saliency, including a new state-of-the-art in video saliency. These results demonstrate that leveraging complementary knowledge across pretrained models yields mutual improvements and scalable benefits as more models participate, with practical implications for leveraging public model repositories in transfer learning. The approach is grounded in a simple, single-stage objective and shows robust improvements even across diverse architectures and training objectives, albeit with limitations related to task-specific supervision and potential constraints in unsupervised settings.

Abstract

Pretrained models are ubiquitous in the current deep learning landscape, offering strong results on a broad range of tasks. Recent works have shown that models differing in various design choices exhibit categorically diverse generalization behavior, resulting in one model grasping distinct data-specific insights unavailable to the other. In this paper, we propose to leverage large publicly available model repositories as an auxiliary source of model improvements. We introduce a data partitioning strategy where pretrained models autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge. Experiments across various tasks demonstrate the effectiveness of our proposed approach. In image classification, we improved the performance of ViT-B by approximately 1.4% through bidirectional knowledge transfer with ViT-T. For semantic segmentation, our method boosted all evaluation metrics by enabling knowledge transfer both within and across backbone architectures. In video saliency prediction, our approach achieved a new state-of-the-art. We further extend our approach to knowledge transfer between multiple models, leading to considerable performance improvements for all model participants.

Figures (3)

• Figure 1: The figure illustrates the training process for a given batch of samples. (a) KD hinton2015distilling utilizes a pretrained teacher model and trains a student model to mimic the teacher's predictions on each sample. (b) KL+DP roth2024fantastic employs a frozen teacher and the original frozen student (called student-teacher) to jointly guide the training of the student model. They partition the dataset into samples where learning from the teacher is desired and ones where knowledge from the original frozen student should be retained. (c) In Bi-KD (ours), both models are trainable and learn from each other on every sample, resulting in bidirectional knowledge transfer. Different from KD and KL+DP, we are able to improve both models simultaneously.
• Figure 2: Generalization of our proposed method for transferring knowledge between multiple pretrained models. For each sample, we select a teacher model that guides the training of all other models. The teacher is selected based on the highest prediction probability, or the lowest loss, corresponding to the ground-truth. This data partitioning strategy enables every model participating in the knowledge transfer to learn from each others strengths and address their weaknesses, resulting in all models improving within a single training stage.
• Figure 3: Canonical Correlation Analysis (CCA) of the Top-100 canonical components between the feature representations of ViT-S and ViT-T. The plot compares correlations from the original models with those from Bi-KD, showing that Bi-KD achieves stronger feature-space alignment.