ELLA: Efficient Lifelong Learning for Adapters in Large Language Models

TL;DR

ELLA tackles catastrophic forgetting in rehearsal-free continual learning for LLM adapters by introducing a subspace-aware regularizer that penalizes alignment with high-energy past update directions, effectively implementing an anisotropic shrinkage of the new task update. The method operates in weight space on LoRA updates and yields a closed-form update that bounds interference, enabling forward transfer while preserving prior knowledge. Empirically, ELLA delivers state-of-the-art replay-free performance across SC, LS, and TRACE, scales robustly from hundreds of millions to multi-billion-parameter models, and improves generalization to unseen tasks with minimal memory and compute overhead. The approach is architecture-agnostic, download-efficient, and easily integrable with existing LoRA-based CL pipelines, offering a principled and scalable solution for lifelong adaptation of LLMs.

Abstract

Large Language Models (LLMs) suffer severe catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing approaches are fundamentally limited: replay-based methods are impractical and privacy-violating, while strict orthogonality-based methods collapse under scale: each new task is projected onto an orthogonal complement, progressively reducing the residual degrees of freedom and eliminating forward transfer by forbidding overlap in shared representations. In this work, we introduce ELLA, a training framework built on the principle of selective subspace de-correlation. Rather than forbidding all overlap, ELLA explicitly characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions, while preserving freedom in the low-energy residual subspaces to enable transfer. Formally, this is realized via a lightweight regularizer on a single aggregated update matrix. We prove this mechanism corresponds to an anisotropic shrinkage operator that bounds interference, yielding a penalty that is both memory- and compute-constant regardless of task sequence length. ELLA requires no data replay, no architectural expansion, and negligible storage. Empirically, it achieves state-of-the-art CL performance on three popular benchmarks, with relative accuracy gains of up to $9.6\%$ and a $35\times$ smaller memory footprint. Further, ELLA scales robustly across architectures and actively enhances the model's zero-shot generalization performance on unseen tasks, establishing a principled and scalable solution for constructive lifelong LLM adaptation.

Figures (6)

• Figure 1: ELLA mitigates interference in continual LoRA training by accumulating past low-rank updates $\mathcal{W}_{\text{past}}$ and applying an energy-based alignment penalty $|| \Delta W_t * \mathcal{W}_{\text{past}} ||_F^2$ to discourage overlap in high-magnitude, task-specific directions. This selective regularization enables parameter reuse in less-used subspaces, achieving a better trade-off between plasticity and stability without requiring task labels, data replay, or architectural modifications.
• Figure 2: Performance impact on Order $4$ (left) and $6$ (right) in terms of BWT. We demonstrate superior resistance to performance decline than baselines (higher values indicate better retention of prior task performance).
• Figure 3: Performance comparison across different backbone size and model families.
• Figure 4: Histogram of prediction loss changes after training on a new task. The ELLA constraint helps reduce the changes -- preserve the loss of previous tasks -- in comparison to when it is not present ($\lambda = 0$).
• Figure 5: Opposing direction weight change across task sequence for T$5$-Large (left) and LLaMA-$3.1$$8$B (right). ELLA consistently reduces backward-conflicting updates, promoting stable continual adaptation.

Theorems & Definitions (2)

• Proposition 1
• proof