DTL: Disentangled Transfer Learning for Visual Recognition

TL;DR

DTL tackles the high memory cost of fine-tuning large pre-trained vision models by disentangling trainable parameters from the backbone. It introduces a Compact Side Network (CSN) of $N$ low-rank mappings with rank $d'$, which aggregates task-specific information and injects it back into later backbone blocks starting at block $M$, while only CSN and the classifier are updated (with $d' \ll d$). An enhanced variant, DTL+, adds a shared global DWConv layer to better capture spatial information. Across VTAB-1K, few-shot, and domain generalization tasks, DTL/DTL+ achieve state-of-the-art or competitive accuracy with substantially fewer trainable parameters and memory footprint than prior PETL methods, enabling practical transfer learning for large models. The approach also enables potential feature reuse across tasks, reducing redundant computation while maintaining strong performance in diverse settings.$

Abstract

When pre-trained models become rapidly larger, the cost of fine-tuning on downstream tasks steadily increases, too. To economically fine-tune these models, parameter-efficient transfer learning (PETL) is proposed, which only tunes a tiny subset of trainable parameters to efficiently learn quality representations. However, current PETL methods are facing the dilemma that during training the GPU memory footprint is not effectively reduced as trainable parameters. PETL will likely fail, too, if the full fine-tuning encounters the out-of-GPU-memory issue. This phenomenon happens because trainable parameters from these methods are generally entangled with the backbone, such that a lot of intermediate states have to be stored in GPU memory for gradient propagation. To alleviate this problem, we introduce Disentangled Transfer Learning (DTL), which disentangles the trainable parameters from the backbone using a lightweight Compact Side Network (CSN). By progressively extracting task-specific information with a few low-rank linear mappings and appropriately adding the information back to the backbone, CSN effectively realizes knowledge transfer in various downstream tasks. We conducted extensive experiments to validate the effectiveness of our method. The proposed method not only reduces a large amount of GPU memory usage and trainable parameters, but also outperforms existing PETL methods by a significant margin in accuracy, achieving new state-of-the-art on several standard benchmarks. The code is available at https://github.com/heekhero/DTL.

Figures (4)

• Figure 1: Top-1 accuracy on VTAB-1K vtab vs. different numbers of trainable parameters and GPU memory footprint. Our DTL achieves the highest accuracy with the least trainable parameters and GPU memory usage.
• Figure 2: Illustration of DTL's network architecture for ViT vit. Our Compact Side Network (CSN) with scarce trainable parameters is plugged in parallel to the backbone blocks. Specifically, before the forward calculation in each block, a low-rank linear mapping lora is applied to the input features to aggregate task-specific side information (orange arrows). This side information is added back to the output of later backbone blocks (green arrows) for adapting backbone features to downstream tasks. During fine-tuning, only parameters of the CSN module and the task-specific classification head are updated (illustrated in orange). Best viewed in color.
• Figure 3: Top-1 accuracy on fine-grained few-shot benchmark with ViT-B/16 as the backbone. Best viewed in color. Note that our approach with less trainable parameters and GPU memory footprint outperforms all baseline methods.
• Figure 4: Top-1 accuracy and peak GPU memory footprint under various $M$ in Eq. \ref{['eq:dtl+_full']}. Our method is consistently effective across different $M$.