Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning

TL;DR

The paper tackles catastrophic forgetting in continual learning for large language models by introducing an adaptive SVD-based full-fine-tuning framework. It decomposes each weight matrix into high-rank (critical) and low-rank (adaptable) subspaces and constrains gradient updates to the low-rank subspace orthogonal to prior high-rank directions, allowing full parameter updates without per-task memory growth. The approach is guided by layer-wise importance derived from input–output similarity and adaptive rank selection, with theoretical justification via a second-order loss expansion and empirical validation on T5-Large, LLaMA-2 7B, and TRACE benchmarks, achieving state-of-the-art results against parameter-efficient baselines while preserving general capabilities. This work provides a principled, scalable solution for continual learning in LLMs, enabling robust knowledge accumulation without forgetting in realistic deployment scenarios.

Abstract

Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones. Existing methods typically rely on low-rank, parameter-efficient updates that limit the model's expressivity and introduce additional parameters per task, leading to scalability issues. To address these limitations, we propose a novel continual full fine-tuning approach leveraging adaptive singular value decomposition (SVD). Our method dynamically identifies task-specific low-rank parameter subspaces and constrains updates to be orthogonal to critical directions associated with prior tasks, thus effectively minimizing interference without additional parameter overhead or storing previous task gradients. We evaluate our approach extensively on standard continual learning benchmarks using both encoder-decoder (T5-Large) and decoder-only (LLaMA-2 7B) models, spanning diverse tasks including classification, generation, and reasoning. Empirically, our method achieves state-of-the-art results, up to 7% higher average accuracy than recent baselines like O-LoRA, and notably maintains the model's general linguistic capabilities, instruction-following accuracy, and safety throughout the continual learning process by reducing forgetting to near-negligible levels. Our adaptive SVD framework effectively balances model plasticity and knowledge retention, providing a practical, theoretically grounded, and computationally scalable solution for continual learning scenarios in large language models.

Figures (13)

• Figure 1: Overview of our Adaptive SVD-based Continual Fine-tuning Method. For each parameter matrix in the network, we perform SVD decomposition to identify high-rank components (associated with larger singular values) that encode crucial knowledge from previous tasks, and low-rank components (associated with smaller singular values) that contribute minimally to model performance. When learning a new task, gradient updates are projected onto the low-rank subspace orthogonal to previous task representations, allowing full parameter updates while minimizing catastrophic forgetting.
• Figure 2: Leaderboard performance impact of low-rank approximations applied to the attn.v_proj.weight (value projection matrix) across selected layers of Granite 8B.
• Figure 3: Leaderboard performance after low-rank approximations of the mlp.gate_proj.weight (first feedforward projection) across layers.
• Figure 4: Effect of low-rank approximation on the mlp.down_proj.weight (third feedforward projection) for later layers in Granite 8B, evaluated on the Leaderboard benchmark.
• Figure 5: L2 norms of matrix-vector products for each singular vector component in the mlp.down_proj.weight matrix (layer 34, Granite 8B), using inputs from a previously learned task. The clear downward trend confirms that low singular directions have minimal activation for the learned task.

Theorems & Definitions (10)

• Lemma 1: Second-Order Approximation of Catastrophic Forgetting
• proof
• Lemma 2: Block-Diagonal Approximation of the Hessian
• proof
• Lemma 3: Relationship Between Layer Importance and Curvature
• proof
• Theorem 1: Hierarchy of Forgetting Bounds
• proof
• Corollary 1: Forgetting Reduction with Adaptive SVD
• proof