Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation

TL;DR

The paper tackles catastrophic forgetting and degraded robustness during fine-tuning of pretrained vision and vision-language models. It extends Task Adaptive Parameter Sharing (TAPS) by introducing an indicator-based gate for Low-Rank Adaptation (LoRA) blocks, enabling selective activation of a small subset of parameters ($W_ ext{ell} = W_{0, ext{ell}} + I_ au(s_i) A_ ext{ell} B_ ext{ell}$) with sparsity encouraged by $\,\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{original}} + \lambda \sum_i |s_i|$. This approach preserves zero-shot and out-of-distribution performance while achieving substantial inference-time speedups (up to $2.9\times$ for LoRA and $5\times$ for DoRA) and using as little as $5\%$ of active blocks. The method is demonstrated on CLIP and DINO-ViT, showing broad applicability across PEFT variants and tasks, including classification and retrieval, with improved robustness under distribution shifts. Overall, the indicator-gated PEFT provides a practical, memory-efficient strategy for domain adaptation that minimizes knowledge loss and maintains generalization.

Abstract

Adapting deep learning models to new domains often requires computationally intensive retraining and risks catastrophic forgetting. While fine-tuning enables domain-specific adaptation, it can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance. Pre-trained zero-shot models like CLIP offer strong generalization but may suffer degraded robustness after fine-tuning. Building on Task Adaptive Parameter Sharing (TAPS), we propose a simple yet effective extension as a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks. Our approach minimizes knowledge loss, retains its generalization strengths under domain shifts, and significantly reduces computational costs compared to traditional fine-tuning. We demonstrate that effective fine-tuning can be achieved with as few as 5\% of active blocks, substantially improving efficiency. Evaluations on pre-trained models such as CLIP and DINO-ViT demonstrate our method's broad applicability and effectiveness in maintaining performance and knowledge retention.

Figures (11)

• Figure 1: Comparison of pretrained CLIP, fully fine-tuned CLIP (FLYP), and LoRA fine-tuning with and without our method (rank 128) on ImageNet-1K (ID). Fine-tuning with FLYP and LoRA results in an average drop of 16.24% and 17.12% in zero-shot task performance (ZS-RET and ZS-CLS), respectively. In contrast, our method activates only 6.25% of the LoRA blocks, reducing catastrophic forgetting by an average of 7.31%, enhancing out-of-distribution (OOD) robustness, and improving inference efficiency. See Section \ref{['sec:vision-language']}.
• Figure 2: Overview of our proposed method incorporating LoRA with an indicator function. This approach allows flexible application of selective layer activation, making it compatible with various low-rank approximation techniques beyond LoRA.
• Figure 3: The figure compares the top-1 accuracy of pretrained and fine-tuned DINO ViT/S-16 models on CIFAR-10 and CIFAR-100 using LoRA and LoRA+Ours across various $\lambda$ values and different ranks. It demonstrates that our method achieves competitive performance on target datasets while preserving prior knowledge. Additionally, as $\lambda$ increases, the percentage of activated blocks decreases.
• Figure 4: The figure compares top-1 accuracy of fine-tuned DINO ViT/S-16 models across transfer datasets and ImageNet-100. It demonstrates that models fine-tuned with LoRA, as well as those incorporating the indicator function, achieve high accuracy on target datasets (e.g., CIFAR-10) while retaining knowledge from the pre-trained dataset.
• Figure 5: Comparison of top-1 accuracy for fine-tuned DINO ViT/B-16 models on CIFAR-100 and ImageNet-100. The results indicate that our method achieves similar performance regardless of model size.