Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

TL;DR

The paper addresses the challenge of transferring complementary knowledge between arbitrary pretrained models trained on the same data, without relying on external rankings. It reframes knowledge transfer as a continual-learning problem and introduces data-partitioned distillation (KL-Dist+DP) that selectively teaches certain samples from a teacher while retaining the student’s prior knowledge. The method yields strong gains across a wide range of model pairs and even enables unsupervised transfer that approaches supervised performance, significantly surpassing standard distillation in many settings. This approach enables effective fusion of knowledge from publicly available model zoos, offering a practical pathway to improve large models with context from weaker or diverse predecessors while maintaining efficiency and without additional supervision.

Abstract

Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -- independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation -- a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock auxiliary gains and knowledge fusion from any model repository without restrictions on model and problem specifics - including from weaker, lower-performance models. This work therefore provides an initial, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a much more general extension through data partitioning for successful transfer between nearly all pretrained models, which we show can also be done unsupervised. Finally, we assess both the scalability and impact of fundamental model properties on successful model-agnostic knowledge transfer.

Figures (11)

• Figure 1: (a) We show the share of complementary knowledge ($\rho^\text{pos}$, perc. of pos. label flips of teacher w.r.t. student) against perform. differences $\Delta_\text{acc}$ for 466 teacher/student pairs, and find significant complementary context even for much weaker teachers. (b) Looking at the entropy of the compl. knowledge distribution over classes, we find context to be more specialized for weaker teachers.
• Figure 2: (a) Sorted histograms of complementary knowledge per class (positive prediction flips) as a share of total class samples for three teacher-student pairs (weak, equal and strong teacher). Complementary context is centralized around few classes. (b) Semantic similarity between top-X$\%$ classes sorted by complementary knowledge amount. Shown as relative difference to average class similarity. Classes with the most complementary knowledge are likely semantically related.
• Figure 3: For general knowledge transfer between any pretrained models, we propose data-level regularization: Samples are separated based on if they should be taught through the teacher $f_t$ or retained via a frozen version of the initial student, $f_\text{st}$. All models forward the same batch, and outputs $\sigma(z_t)$ and $\sigma(z_{st})$ are merged on a sample-level via selection masks $m_t$ and $m_{st}$ derived from model confidences (Eq. \ref{['eq:supervised_masks']}). Lastly, we compute the KL-div. to the adapting student's ($f_s$) outputs $\sigma(z_{s})$.
• Figure 4: (a) Share of teachers resulting in positive knowledge transfer (success rate) for knowledge distillation variants (blue) and continual learning extensions (orange). Each box represents 400 transfer experiments, with a clear increase for continual learning setups. (b) Transfer delta by binned teacher-student performance difference. For more robust reporting, we show the mean transfer delta of the top 25% for each bin/approach with the same 400 teacher-student pairs. The results show KL-Dist+DP Transfer enabling consistent gains from weaker and stronger teachers.
• Figure 5: (a) Transfer rates for the sets of classes containing the top 2% - 100% complementary knowledge per model family. (b) Transfer rates versus student size separated by model family.