Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach

TL;DR

This work tackles the memory burden of parameter-efficient transfer learning for Vision Transformers by proposing SynQT, a disentangled framework that separates task-specific learning from pre-trained knowledge. A lightweight Query Synthesis Module (QSM) generates a task-driven query, while a frozen Knowledge Extraction Module (KEM) leverages intermediate representations to extract useful features, feeding a sophisticated, instance-aware classification head. Across VTAB-1K and FGVC few-shot benchmarks, SynQT achieves state-of-the-art memory efficiency and competitive accuracy, often outperforming strong PETL baselines under memory constraints. The approach is versatile, scalable to different backbones, and supported by extensive ablations and visual analyses demonstrating the value of disentanglement for memory-conscious transfer learning in vision models.

Abstract

Recent works on parameter-efficient transfer learning (PETL) show the potential to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. However, since they usually insert new structures into the pre-trained model, entire intermediate features of that model are changed and thus need to be stored to be involved in back-propagation, resulting in memory-heavy training. We solve this problem from a novel disentangled perspective, i.e., dividing PETL into two aspects: task-specific learning and pre-trained knowledge utilization. Specifically, we synthesize the task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. The synthesized query equipped with task-specific knowledge serves to extract the useful features for downstream tasks from the intermediate representations of the pre-trained model in a query-only manner. Built upon these features, a customized classification head is proposed to make the prediction for the input sample. lightweight architecture and avoids the use of heavy intermediate features for running gradient descent, it demonstrates limited memory usage in training. Extensive experiments manifest that our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.

Figures (9)

• Figure 1: Overall architecture of SynQT. Each block in SynQT contains two key designs, including Query Synthesis Module (QSM) and Knowledge Extractor Module (KEM). QSM is a lightweight learnable module to capture task-specific knowledge and KEM reuses frozen weights in the pre-trained model for feature extraction. The rich and hierarchical features from KEMs are fed into the classification head.
• Figure 2: Classification head in SynQT, corresponding to the head in Figure \ref{['fig:framework']}. We utilize a shared projection to align the hierarchical features and apply feature aggregation conditioned on the output of the last block $H_l$. We also adopt DropFeat to prevent overfitting, which randomly drops some features during training.
• Figure 3: Performance comparison under memory constraints. SynQT outperforms other methods with different memory costs.
• Figure 4: Performance comparison in few-shot learning. Overall, SynQT outperforms other baselines in the extreme low-data regime.
• Figure 5: Combining different PETL methods. AF is short for AdaptFormer.