Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification

TL;DR

ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters, is proposed and it is demonstrated that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27% average harmonic mean accuracy.

Abstract

Recent advances in fine-tuning Vision-Language Models (VLMs) have witnessed the success of prompt tuning and adapter tuning, while the classic model fine-tuning on inherent parameters seems to be overlooked. It is believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. In this paper, we revisit this viewpoint, and propose a new perspective: fine-tuning the specific parameters instead of all will uncover the power of classic model fine-tuning on VLMs. Through our meticulous study, we propose ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27\% average harmonic mean accuracy. Lastly, to understand how fine-tuning in CLIPFit affects the pre-trained models, we conducted extensive experimental analyses w.r.t. changes in internal parameters and representations. We found that low-level text bias layers and the first layer normalization layer change much more than other layers. The code is available at \url{https://github.com/minglllli/CLIPFit}.

Figures (11)

• Figure 1: Comparison of (a) prompt tuning methods, (b) adapter tuning methods, and (c) our proposed CLIPFit method. Prompt tuning methods introduce a set of learnable external parameters as input to learn task-specific knowledge. Adapter tuning methods introduce extra learnable networks following the image encoder to learn task-specific features. Unlike these two methods, our CLIPFit does not introduce external parameters and fine-tunes only a small portion of the CLIP model.
• Figure 2: An overview of our CLIPFit. Unlike existing prompt tuning methods or adapter tuning methods, CLIPFit does not introduce any external parameters and fine-tunes specific inherent parameters of CLIP. For the text encoder, as shown in the upper part of the figure, CLIPFit fine-tunes only the bias terms of projection linear layers in feed-forward networks. For the image encoder, as shown in the lower part of the figure, CLIPFit updates LayerNorm.
• Figure 3: Visualization of changes in different layers.
• Figure 4: Left: visualization of squared gradient sum. Right: visualization of change w/ regularization loss.
• Figure 5: Visualization of learned image feature space from zero-shot CLIP and CLIPFit via t-SNE.