Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification

TL;DR

The paper tackles the problem of fine-tuning vision-language models with very limited data by addressing catastrophic forgetting when applying LoRA. It introduces Comp-LoRA, a method that constrains the low-rank update to the complementary subspace of pretrained weight directions, thereby preserving core vision-language alignment while learning new task-specific information. The authors provide theoretical and empirical justification, including an optimization view and extensive experiments on 11 datasets showing improved Top-1 accuracy and better zero-shot preservation compared to baselines. The findings highlight the importance of subspace-aware regularization and offer guidance on selecting the complementary subspace dimension, with practical implications for efficient, robust few-shot VLM adaptation. The approach is compatible with other regularization strategies and advances the state of the art in few-shot CLIP-style fine-tuning.

Abstract

Vision language model (VLM) has been designed for large scale image-text alignment as a pretrained foundation model. For downstream few shot classification tasks, parameter efficient fine-tuning (PEFT) VLM has gained much popularity in the computer vision community. PEFT methods like prompt tuning and linear adapter have been studied for fine-tuning VLM while low rank adaptation (LoRA) algorithm has rarely been considered for few shot fine-tuning VLM. The main obstacle to use LoRA for few shot fine-tuning is the catastrophic forgetting problem. Because the visual language alignment knowledge is important for the generality in few shot learning, whereas low rank adaptation interferes with the most informative direction of the pretrained weight matrix. We propose the complementary subspace low rank adaptation (Comp-LoRA) method to regularize the catastrophic forgetting problem in few shot VLM finetuning. In detail, we optimize the low rank matrix in the complementary subspace, thus preserving the general vision language alignment ability of VLM when learning the novel few shot information. We conduct comparison experiments of the proposed Comp-LoRA method and other PEFT methods on fine-tuning VLM for few shot classification. And we also present the suppression on the catastrophic forgetting problem of our proposed method against directly applying LoRA to VLM. The results show that the proposed method surpasses the baseline method by about +1.0\% Top-1 accuracy and preserves the VLM zero-shot performance over the baseline method by about +1.3\% Top-1 accuracy.

Figures (16)

• Figure 1: The architecture of Comp-LoRA with comparison to original LoRA. We first project the input $x$ into the complementary subspace using the pre-computed matrix $U^c$ and also project the output back to the hidden subspace by matrix $V^c$. In the complementary subspace, we also leverage the LoRA architecture for efficient optimization.
• Figure 2: SVD decomposition for weights in linear layers. We eliminate the top-k most effective directions and obtain the complementary subspace that is represented by the rest directions.
• Figure 3: The diagram of complementary subspace. We optimize the LoRA module in the subspace that is complemented to the principal directions of the pretrained weight matrix.
• Figure 4: The comparison experiments of different methods on the average accuracy score. The proposed method Comp-LoRA outperforms other methods.
• Figure 5: The effect of complementary subspace dimension on ImageNet. The X-axis is scaled for better demonstration. The performance of the proposed Comp-LoRA firstly increases and then decreases when the complementary subspace dimension decreases, as far as the main trend. Within the range of $[511, 384]$, the proposed Comp-LoRA outperforms the baseline CLIP-LoRA method.

Theorems & Definitions (3)

• Definition 3.1: Complementary Subspace
• Remark
• Definition 3.2: Subspace Projection