SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

TL;DR

This work proposes Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal, resulting in an excellent stability-plasticity trade-off.

Abstract

Continual Learning (CL) with foundation models has recently emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. However, existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. To address these limitations, we propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends to follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting in an excellent stability-plasticity trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAs can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component selection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code is available at https://github.com/WuYichen-97/SD-Lora-CL.

Figures (7)

• Figure 1: Illustration of the parameter update in (a) Vanilla LoRA and (b) the proposed SD-LoRA, where the current task index is $t=2$ and $r, r_1, r_2 \ll\min\{m,n\}$.
• Figure 2: (a) Distances between the five optimal weights $\{\mathbf{W}_i^\star\}$ on ImageNet-R ($N=5$) relative to the foundation model weights $\mathbf{W}_0$. All relative distances are much smaller than one, indicating that $\{\mathbf{W}^\star_i\}$ are closer to each other than to $\mathbf{W}_0$. (b) and (c) Performance comparison of Vanilla LoRA versus LoRA with the first learned direction fixed, on ImageNet-R across five and ten tasks, respectively. Shaded regions indicate standard error.
• Figure 3: Analysis of the learning process of SD-LoRA. (a) Least squares fitting residual between the newly learned direction $\overline{\mathbf{A}_t\mathbf{B}_t}$ and all previous directions $\{\overline{\mathbf{A}_k\mathbf{B}_k}\}_{k=1}^{t-1}$ over time. (b) and (c) Learned magnitudes $\{\alpha_k\}_{k=1}^{N}$ on ImageNet-R across five and ten tasks, respectively.
• Figure 4: Learning trajectory comparison of vanilla LoRA and SD-LoRA. (a) Toy illustration of the learning trajectories for vanilla LoRA ($\mathbf{W}_0 \rightarrow {\mathbf{W}}_1^{\textrm{LoRA}} \rightarrow {\mathbf{W}}_2^{\textrm{LoRA}}$) and SD-LoRA ($\mathbf{W}_0 \rightarrow {\mathbf{W}}_1^{\textrm{SD}} \rightarrow {\mathbf{W}}_2^{\textrm{SD}}$) across two sequential tasks. (b) Classification accuracy along the vanilla LoRA path. The improvement on $\mathcal{T}_2$ but degradation on $\mathcal{T}_1$ indicates that vanilla LoRA suffers from catastrophic forgetting. (c) Classification accuracy along the SD-LoRA path, showing that it successfully lands on an overlapping low-loss region.
• Figure 5: Average accuracy during sequential training on (a) ImageNet-A ($N=10$), (b) ImageNet-R ($N=10$), and (c) ImageNet-R ($N=5$) using ViT-B/16 from DINO caron2021emerging.

Theorems & Definitions (3)

• Theorem 1
• Theorem 2
• proof