Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation

TL;DR

The paper tackles catastrophic forgetting in continual learning by proposing PEARL, a rehearsal-free, parameter-efficient framework that dynamically allocates low-rank adapters (LoRA) per task based on the proximity of current task weights to reference task weights. By computing a task vector via SV decomposition and using a dynamic threshold to determine rank, PEARL reuses information from a reference task while adapting minimally for new tasks. Extensive experiments across ResNet, BSC, and ViT backbones in Class-IL and Task-IL settings demonstrate that PEARL outperforms baselines with modest parameter growth. The work advances practical continual learning for small to mid-sized models, though it requires task boundaries and incurs some inference overhead in Class-IL scenarios.

Abstract

Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with a lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low rank adapters (LoRA) in these approaches are highly sensitive to rank selection which can lead to sub-optimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific LoRA components based on the current tasks' proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate it across three vision architectures (ResNet, Separable Convolutional Network and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin.

Figures (5)

• Figure 1: Visualization of final task dynamic resource allocation in Seq-CIFAR100 (5T). (left) Dynamic threshold of different convolutional layers based on the similarity of current task with respect to reference task for deciding the rank. (middle) Selected rank based on dynamic threshold. (right) Actual rank of re-shaped 2-dimensional weight matrix. As can be seen both dynamic threshold and selected rank show more emphasis for later layers, in line with the actual rank.
• Figure 2: Task-IL performance comparison with parameter isolation approaches on Seq-CIFAR100 (20T). We report the final Task-IL accuracies of each task after training on all tasks. Methods compared are NISPA gurbuz2022nispa, PNN rusu2016progressive, PAE hung2019increasingly, CPG hung2019compacting, PackNet mallya2018packnet, and CLNP golkar2019continual
• Figure 3: Evaluation of Class-IL ($A_T$% ) performance under longer task sequences in Seq-CIFAR100. In line with other parameter isolation approaches, there is a steady drop in performance as we move from 5 tasks to 10 tasks and eventually to 20 tasks.
• Figure 4: Depiction of final task dynamic resource allocation in ImageNet-R (5T). (left) Dynamic threshold of different ViT layers based on the similarity of current task with respect to reference task for deciding the rank. (middle) Selected rank based on dynamic threshold. (right) Actual rank of Key weight matrix. As can be seen both dynamic threshold and selected rank show more emphasis for later layers, in line with the actual rank.
• Figure 5: Task-wise resource allocation in parameter space (normalized number of parameters) for PEARL variants. (a) Visualizes resource allocation across Seq-CIFAR100 (10T), Seq-TinyImageNet (10T), and ImageNet-R (10T). (b) Shows resource allocation dynamics for ImageNet-R (10T) using ViT across three runs.

Theorems & Definitions (1)

• Theorem 3.1