KeepLoRA: Continual Learning with Residual Gradient Adaptation

TL;DR

The paper addresses continual learning for vision-language models by revealing that general knowledge is stored in a principal weight subspace while domain-specific adaptations reside in the residual subspace. It proposes KeepLoRA, a gradient-projection-based method that updates only in the residual subspace while initializing from the current-task gradient and maintaining orthogonality to both the pre-trained and prior-task subspaces. Theoretical analysis (via subspace projection and the Eckart–Young–Mirsky theorem) supports optimal balance between plasticity and stability, and extensive experiments on CLIP and LLaVA demonstrate state-of-the-art performance across MTIL, DCL, and UCIT benchmarks. The approach is parameter-efficient, incurs no inference overhead, and is readily applicable to large-scale models, offering a practical solution for continual learning in multimodal AI systems.

Abstract

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

Figures (4)

• Figure 1: Analysis of model parameter subspaces and overall CL performance. In Fig. \ref{['fig:sub_a']} and \ref{['fig:sub_b']}, we measure zero-shot performance after reconstructing attention weights using only the top principal singular components. While performance on general-domain datasets remains highly robust, performance on most specific-domain datasets degrades sharply as more low-energy components are removed. In Fig. \ref{['fig:sub_c']}, the Last metric measures the accuracy gain on the final learned task relative to a zero-shot baseline, while Transfer measures the accuracy degradation on unseen tasks.
• Figure 2: Visualization of the average L2 norm of the output magnitude from the learned LoRA across multiple tasks. Each heatmap cell at row $i$ and column $j$ displays the normalized average L2 norm of the LoRA’s output when the model, trained up to task $i$, is tested on task $j$’s data. The vertical bar to the left of each heatmap indicates the mean output norm across all test tasks after each training stage, with darker colors signifying a lower norm and thus a reduced impact on the stability.
• Figure 3: Comparison of plasticity between KeepLoRA and the LoRA baseline under the same learnable parameter budgets: Fig. \ref{['fig:sub_aa']} 0.49 million parameters and Fig. \ref{['fig:sub_bb']} 0.98 million parameters. Each bar represents the performance drop for a task, measured as the difference between accuracy from isolated training and accuracy after sequential learning and immediate testing.
• Figure 4: Effects of hyperparameters $\epsilon_{w(\text{vision})}$ and $\epsilon_{w(\text{text})}$ on Transfer and Last, respectively.

Theorems & Definitions (4)

• Proposition 3.1
• Proposition 3.2
• proof
• proof : Proof