Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

TL;DR

This work decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold, which mitigates both backward and forward forgetting, consistently outperforming continual learning baselines.

Abstract

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.

Figures (5)

• Figure 1: Comparison of parameter-efficient methods on the UCIT benchmark in terms of MFN and FWT. MFN measures the model’s final performance after learning all tasks, whereas FWT reflects the ability to generalize learned knowledge to unseen tasks. Zero-shot upper bound is the mean accuracy of the base model, while LoRA-first upper bound is the mean accuracy of LoRA on each task immediately after it is learned. EBLoRA clearly outperforms all baselines in MFN and FWT, achieving performance close to both upper bounds.
• Figure 2: Imbalanced low-rank components amplify cross-task interference. (a) LoRA updates exhibit long-tailed singular value spectra where a few components dominate adaptation energy, while variance increases during training. (b) Multi-task LoRA merging reveals that direct merging degrades performance, while singular value smoothing can reduce interference and improve task coexistence.
• Figure 3: Comparison between LoRA-FT and EBLoRA (Ours) on the MLLM-DCL (left) and UCIT (right) benchmarks. The radar plots display the accuracy of each task immediately after it is learned, showing that our method preserves the model's ability to continuously learn new tasks.
• Figure 4: Interference analysis of balanced singular value. Noise is generated through merging LoRA weights trained on UCIT tasks. Left: applying partially smoothed noise on LoRA reduces interference in seen tasks. Right: under equal-norm perturbations, EBO approach exhibits higher robustness compared to LoRA.
• Figure 5: Heatmaps of LoRA (left) and EBO (right) in UCIT tasks. The upper triangle denotes the performance on unseen tasks, while the lower triangle denotes performance on seen tasks. Raw data for these visualizations can be found in Tab. \ref{['tab:ucit_loraft']} and \ref{['tab:EBO_ucit']}.

Theorems & Definitions (4)

• Proposition 3.1: Optimality of the tangent-space projection
• Proposition 3.2: Optimality of the manifold retraction
• proof
• proof