Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

Jianlu Shen, Fu Feng, Jiaze Xu, Yucheng Xie, Jiaqi Lv, Xin Geng

TL;DR

BoT (Bidirectional knowledge Transfer) is proposed, the first size-agnostic framework to unify S2L and L2S scaling and leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner.

Abstract

Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively. This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework. In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling. Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge. This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT). BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.

A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling

TL;DR

BoT (Bidirectional knowledge Transfer) is proposed, the first size-agnostic framework to unify S2L and L2S scaling and leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner.

Abstract

Transferring pre-trained knowledge from a source model to a target model of a different architectural size is a key challenge for flexible and efficient model scaling. However, current parameter-space methods treat Small-to-Large (S2L) and Large-to-Small (L2S) scaling as separate, incompatible problems, focusing on parameter synthesis and selection, respectively. This fragmented perspective has resulted in specialized tools, hindering a unified, bidirectional framework. In this paper, we propose BoT (Bidirectional knowledge Transfer), the first size-agnostic framework to unify S2L and L2S scaling. Our core insight is to treat model weights as continuous signals, where models of different sizes represent distinct discretizations of the transferable knowledge. This multi-resolution perspective directly casts S2L and L2S scaling as the signal processing operations of upsampling and downsampling, naturally leading to the adoption of the Discrete Wavelet Transform (DWT) and its Inverse (IDWT). BoT leverages the recursive nature of wavelets, using the decomposition level as a dynamic scaling factor to bridge disparate model sizes in a parameter-free and computationally efficient manner. Extensive experiments on DeiT, BERT, and GPT demonstrate significant pre-training FLOPs savings (up to 67.1% for S2L, 52.8% for L2S) and state-of-the-art performance on benchmarks like GLUE and SQuAD.
Paper Structure (54 sections, 8 equations, 8 figures, 16 tables)

This paper contains 54 sections, 8 equations, 8 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Initialization
  4. Frequency Analysis in Networks
  5. Methods
  6. Preliminaries
  7. L2S Transfer via Wavelet Decomposition
  8. S2L Transfer via Wavelet Synthesis
  9. Experiments
  10. Experimental Setup
  11. Results on Large-Scale Vision Models
  12. Results on BERT
  13. Results on GPT2
  14. Ablation and Study
  15. Analysis of Different Wavelet Families
  16. ...and 39 more sections

Figures (8)

  • Figure 1: Conceptual illustration of knowledge transfer challenges between mismatched model sizes. (a) Direct fine-tuning from model zoos is often unviable due to size mismatches. (b) Small-to-Large (S2L) methods initialize by mapping or stacking weights from a small model. (c) Large-to-Small (L2S) methods initialize by selecting weights from a large model.
  • Figure 2: Comparison of cross-architecture initialization frameworks. (a) Previous Large-to-Small (L2S) methods rely on heuristic sampling or layer selection. (b) Previous Small-to-Large (S2L) methods use direct copying or trainable mapping functions. (c) Our proposed BOT framework unifies both directions through a parameter-free, frequency-domain transformation using the 3D-DWT and IDWT.
  • Figure 3: Results of pretraining DeiT-S, BERT-S and GPT-S. BoT can achieve the highest savings in FLOPs with 22.0% for DeiT-S, 67.1% for BERT-S, and 58.3% for GPT-S from the Scratch models.
  • Figure 4: Results of pretraining DeiT-B, BRET-B and GPT-B. BoT can achieve the highest savings in FLOPs with 22.0% for DeiT-B, 67.1% for BERT-B, and 58.3% for GPT-B from the Scratch models.
  • Figure 5: Ablation study on the choice of wavelet family. Lower validation loss indicates better performance.
  • ...and 3 more figures