Merge before Forget: A Single LoRA Continual Learning via Continual Merging

TL;DR

The paper introduces SLAO, a continual learning method for LLMs that merges every task’s LoRA update into a single shared LoRA. SLAO uses orthogonal initialization to minimize interference across tasks and a time-aware merging strategy that respects LoRA component asymmetry, achieving constant memory usage with respect to the number of tasks. The authors provide NTK-informed theoretical support and demonstrate through extensive experiments on Llama and Qwen models that SLAO improves accuracy and efficiency across standard CL benchmarks, large task sets, and the SuperNI suite. This approach offers a scalable, data-free path to resilient continual learning in large language models with practical impact for deployment in evolving task environments.

Abstract

Parameter-efficient continual learning has emerged as a promising approach for large language models (LLMs) to mitigate catastrophic forgetting while enabling adaptation to new tasks. Current Low-Rank Adaptation (LoRA) continual learning techniques often retain and freeze previously learned LoRAs or generate data representations to overcome forgetting, typically utilizing these to support new LoRAs learn new tasks. However, these methods not only ignore growing computational memory with tasks and limited storage space but also suffer from potential task interference due to the lack of effective LoRA merging mechanisms. In this paper, we propose a novel continual learning method that orthogonally initializes and sequentially merges LoRAs updates into a single unified LoRA. Our method leverages orthogonal basis extraction from previously learned LoRA to initialize the learning of new tasks, further exploits the intrinsic asymmetry property of LoRA components by using a time-aware scaling mechanism to balance new and old knowledge during continual merging. Our approach maintains constant memory complexity with respect to the number of tasks, minimizes interference between past and new tasks via orthogonal basis initialization, and improves performance over asymmetric LoRA merging via adaptive scaling. We provide theoretical analysis to justify our design and conduct extensive experiments across diverse continual learning benchmarks using various Llama models, demonstrating the effectiveness and efficiency of our method.

Figures (5)

• Figure 1: Cosine similarity between 15 tasks from the large number of tasks benchmark for fine-tuned q and v attention LoRA $\boldsymbol{A}$ and $\boldsymbol{B}$ in the last layer (32nd) of Llama-2-7B-chat.
• Figure 2: Comparison of SLAO and O-LoRA memory usage of large number of tasks benchmark via Llama-2-7B-chat.
• Figure 3: Overview of SLAO. Left area is a framework where fine-tuned LoRA (orange) and merged LoRA (gray) are processed over time. Right area (2 blue boxes) highlights key components: (1) Orthogonal initialization for new task $i+1$ learning LoRA, where orthogonal basis is extracted from $\boldsymbol{A}_{\text{ft},i}$ to initialize $\boldsymbol{A}_{\text{ft},i+1}$ such that $\boldsymbol{A}_{\text{ft},i+1}^{(0)}(\boldsymbol{A}_{\text{ft},i+1}^{(0)})^{\top}=\boldsymbol{I}_r$ and $\boldsymbol{B}_{\text{ft},i+1}$ is initialized by previous $\boldsymbol{B}_{\text{ft},i}$; (2) Time-aware continual merging for $\boldsymbol{B}_{\text{ft},i+1}$ and $\boldsymbol{B}_{\text{merge}}^{i}$, and update $\boldsymbol{A}_{\text{merge}}^{i+1}$ via $\boldsymbol{A}_{\text{ft},i+1}$.
• Figure 4: Comparison of model performance across time coefficients.
• Figure 5: Cosine similarity between 15 tasks from SuperNI benchmark for fine-tuned q and v attention LoRA $\boldsymbol{A}$ and $\boldsymbol{B}$ in the last layer (32nd) of Llama-2-7B-chat.

Theorems & Definitions (5)

• Theorem 1
• Lemma 1
• proof
• Theorem 2
• proof