Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Models

TL;DR

A novel MTIL framework, named AFA, is proposed, which comprises two core modules: an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs and an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning.

Abstract

This study aims to address the problem of multi-domain task incremental learning~(MTIL), which requires that vision-language models~(VLMs) continuously acquire new knowledge while maintaining their inherent zero-shot recognition capability. Existing paradigms delegate the testing of unseen-domain samples to the original CLIP, which only prevents the degradation of the model's zero-shot capability but fails to enhance the generalization of the VLM further. To this end, we propose a novel MTIL framework, named AFA, which comprises two core modules: (1) an against forward-forgetting adapter that learns task-invariant information for each dataset in the incremental tasks to enhance the zero-shot recognition ability of VLMs; (2) an against backward-forgetting adapter that strengthens the few-shot learning capability of VLMs while supporting incremental learning. Extensive experiments demonstrate that the AFA method significantly outperforms existing state-of-the-art approaches, especially in few-shot MTIL tasks, and surpasses the inherent zero-shot performance of CLIP in terms of transferability. The code is provided in the Supplementary Material.

Figures (4)

• Figure 1: Comparison between existing work and our approach:(a) The existing MTIL paradigm. (b) Our proposed paradigm.
• Figure 2: Overview of AFA method. (a) AFFA module (left) is trained on $D^t$ using $\mathcal{L}_{\text{affa}}$ to boost the generalization of VLMs. (b) the AFBA module (right) expands a task-specific router $h^t$ at stage $t$ to train $D^t$ while freezing $h^1,\ldots,h^{t-1}$ to prevent catastrophic forgetting. In addition, the expert component comprises multi-head LoRA, designed to enhance the model's few-shot learning ability.
• Figure 3: Impact of numbers of experts and top-k activated experts. The experiment is conducted under the MTIL setting.
• Figure 4: Impact of different Numbers of KNN-N and the threshold. The experiment is based on full-shot MTIL. If the distribution score of the test sample is below the threshold, it is classified as belonging to an unseen domain and processed by the AFFA module.