PLAN: Proactive Low-Rank Allocation for Continual Learning

TL;DR

PLAN tackles continual learning by extending LoRA with proactive low-rank subspace allocation. It represents task updates as $W_t = W_{t-1} + B_t A_t$, selects $A_t$ from a fixed orthogonal basis, and optimizes $B_t$ under a forward-looking min-max objective that anticipates interference along the unallocated subspace $M_t$. A perturbation-based selection rule then guides the next task’s subspace, achieving robust knowledge retention with efficient storage. Empirical evaluations on CIFAR-100, DomainNet, and ImageNet-R show PLAN attaining state-of-the-art results across both accuracy and stability, while remaining compatible with both supervised and self-supervised pretraining of foundation models.

Abstract

Continual learning (CL) requires models to continuously adapt to new tasks without forgetting past knowledge. In this work, we propose \underline{P}roactive \underline{L}ow-rank \underline{A}llocatio\underline{N} (PLAN), a framework that extends Low-Rank Adaptation (LoRA) to enable efficient and interference-aware fine-tuning of large pre-trained models in CL settings. PLAN proactively manages the allocation of task-specific subspaces by introducing orthogonal basis vectors for each task and optimizing them through a perturbation-based strategy that minimizes conflicts with previously learned parameters. Furthermore, PLAN incorporates a novel selection mechanism that identifies and assigns basis vectors with minimal sensitivity to interference, reducing the risk of degrading past knowledge while maintaining efficient adaptation to new tasks. Empirical results on standard CL benchmarks demonstrate that PLAN consistently outperforms existing methods, establishing a new state-of-the-art for continual learning with foundation models.

Figures (4)

• Figure 1: Conceptual illustration of PLAN compared to existing approaches. (a) Vanilla sequential training (orange arrow) causes parameter interference, moving the model away from previously low-loss regions. Existing methods (green arrow) passively avoid interference by enforcing orthogonality, typically assuming simplified low-loss regions for each task. (b) In practice, tasks possess multiple optimal regions. PLAN (purple arrow) is the first method to proactively optimize task-specific subspaces, explicitly anticipating future interference and robustly preserving performance within favorable regions for both current and previous tasks.
• Figure 2: Overview of the proposed PLAN method. (a) Selection mechanism for $A_t$. For clarity, the case $t=1$ is shown, where the current basis set $\mathcal{E}_0$ is the complete standard orthogonal basis. (b) Optimization objective for $B_t$ using worst-case perturbations along unselected basis vectors, i.e. those not included in $A_t$, (denoted by $\mathcal{E}_t = set(M_t)$), with the gradient update directions visually indicated.
• Figure 3: Variation of the performance of different methods during the learning of ImageNet-R and CIFAR100.
• Figure 4: Top-10 selected basis indices (y-axis) for the next task as a function of the sliding window size $S$ (x-axis). Each colored line tracks a specific basis index. The selection stabilizes with a small $S$.