CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

TL;DR

This work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning.

Abstract

Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability, pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at \url{https://github.com/srvCodes/clap4clip}.

Figures (17)

• Figure 1: Concept diagram for probabilistic finetuning of pre-trained CLIP in a CL setup: We identify four such suitable design choices for probabilistic modelling. Choice $\#1$derakhshani2023variational performs variational modelling by imposing prior on the prompt space which makes it prompt-type dependent while also interfering with the in-domain knowledge learning capability of the prompts -- a criterion crucial in the deployment of CL models. Choice $\#2$ (see Sec. \ref{['subsec:feature_var_model']}) instead imposes a prior on the outputs of the text encoder. While this makes it prompt-type agnostic (as the text features can now be derived from arbitrary prompt types), not taking the visual information into account nevertheless leads to the loss of information about cross-modal interactions between the visual and textual cues -- a property essential for preventing cross-modal deviation of finetuned features in CL (see Sec. \ref{['sec:vga']}). Choice $\#3$ (Ours) leverages the best of both worlds by modelling the distribution of visual-guided text features. To further refine the learned distributions of CL tasks, we finally introduce lightweight task-specific adapter modules in choice $\#4$ that make the cross-task centroids more distinct while preserving the aforesaid properties (see Sec. \ref{['sec:task_sp_encoders']}).
• Figure 2: CLAP$\mathbf{4}$CLIP overview: the visual-guided attention (VGA) inference module uses the text features as query (Q), and the visual features as keys (K) and values (V) to produce visual-guided text features. The task-specific text features are fed to their respective task distribution encoders $(\mu^t, \sigma^t)$. The task distribution samples are then fused with the original task features prior to deriving the task logits $y^t$. All task logits are concatenated to produce the final prediction $y^{1:t}$.
• Figure 3: Need for Visual-guided Attention (VGA) inference module. Fig. \ref{['fig:alpha_effect']}: A simple adapter is inadequate at preventing catastrophic forgetting in CL -- marked by high BwT scores; Fig. \ref{['fig:rotation_angle']}: VGA module encourages cross-modal alignment between the learned text features and the pre-trained visual features -- marked by a decrease in average angle $\texttt{arccos} \langle t, 1 \rangle$ between them -- where otherwise the former deviates further with incremental training steps.
• Figure 4: Need for task-specific probabilistic adapters: Cosine distance between the centroids of class-specific latent variables produced without (left) and with (right) task-specific adapters on CIFAR100 (10 tasks, 10 classes per task).
• Figure 5: Parameter count comparison.