Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, Xiaojie Wang

TL;DR

This work introduces Controlled LoRA (CLoRA), a subspace regularization technique for LoRA that constrains the direction of LoRA updates’ null space to mitigate catastrophic forgetting in large language models. By decomposing updates as $\nabla W = AB^{T}$ and applying orthogonal regularization with predefined matrices, CLoRA achieves a favorable balance between preserving base model capacity and limiting output changes during both one-stage fine-tuning and continual learning. Empirical results across commonsense and mathematical benchmarks show CLoRA outperforms baselines in in-domain and out-domain settings and yields strong continual learning performance, with analyses indicating reduced forgetting without overly constraining learning. The method is lightweight, flexible (varying $k$), and complementary to existing PEFT approaches, offering practical gains for deploying robust LLM adapters in evolving tasks.

Abstract

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a sub-space regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix's null space. Experimental results on one-stage LLM finetuning tasks and continual learning settings highlight the superority of CLoRA as a effective parameter efficient finetuning method with catastrophic forgetting mitigating.Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.

Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

TL;DR

This work introduces Controlled LoRA (CLoRA), a subspace regularization technique for LoRA that constrains the direction of LoRA updates’ null space to mitigate catastrophic forgetting in large language models. By decomposing updates as and applying orthogonal regularization with predefined matrices, CLoRA achieves a favorable balance between preserving base model capacity and limiting output changes during both one-stage fine-tuning and continual learning. Empirical results across commonsense and mathematical benchmarks show CLoRA outperforms baselines in in-domain and out-domain settings and yields strong continual learning performance, with analyses indicating reduced forgetting without overly constraining learning. The method is lightweight, flexible (varying ), and complementary to existing PEFT approaches, offering practical gains for deploying robust LLM adapters in evolving tasks.

Abstract

Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a sub-space regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix's null space. Experimental results on one-stage LLM finetuning tasks and continual learning settings highlight the superority of CLoRA as a effective parameter efficient finetuning method with catastrophic forgetting mitigating.Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.

Paper Structure

This paper contains 45 sections, 3 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Mitigating Catastrophic Forgeting
  4. Data-based methods
  5. Architecture-based methods
  6. Learning-based methods
  7. LoRA and Subspace Tuning
  8. Prelimilaries
  9. Notations
  10. Problem Definition
  11. LLM Finetuning
  12. Continual Learning
  13. Method
  14. CLoRA Modeling
  15. Initialization
  16. ...and 30 more sections

Figures (2)

  • Figure 1: Illustration of the intuition behind our approach. For input $x$, the component in $\mathrm{Null}(\Delta W)$ (null space of the updating matrix $\Delta W$) would be ignored, the change of output $\Delta y$ is obtained only from the component in $\mathrm{Row}(\Delta W)$ (row space of $\Delta W$, the orthogonal complement of $\mathrm{Null}(\Delta W)$). CLoRA introduces a pre-defined subset of $\mathrm{Null}(\Delta W)$ by imposing orthogonal regularization with pre-defined matrix $P$.
  • Figure 2: Illustration of CLoRA on typical decoder-only transformer based LLMs. LoRA updating is applied on v-proj in multi-head attention layer for each layer. CLoRA add orthogonal loss computes from trainable LoRA parameters (A and B) to the original language modeling loss.