Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge
Pengwei Tang, Xiaolin Hu, Yong Liu, Lizhong Ding, Dongjie Zhang, Xing Wu, Debing Zhang
TL;DR
The paper argues that preserving pre-trained knowledge during LoRA-based fine-tuning hinges on the initialization space of LoRA adapters rather than the residual weights. It introduces LoRA-Null, which initializes adapters in the null space of pre-trained knowledge activations $X_{ ext{pre}}$, and shows that this space has low effective rank, concentrating information away from pre-trained components. Through extensive experiments on LLaMA-2/3B models across Math, Code, and Instruction tasks, LoRA-Null achieves strong knowledge preservation with competitive downstream performance, outperforming prior methods in preservation on key benchmarks. The work provides theoretical and empirical support for prioritizing initialization-space orthogonality and offers practical guidance for activation-aware LoRA initialization to mitigate forgetting in large language models.
Abstract
Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs), but it still suffers from catastrophic forgetting. Recent work has shown that specialized LoRA initialization can alleviate catastrophic forgetting. There are currently two approaches to LoRA initialization aimed at preventing knowledge forgetting during fine-tuning: (1) making residual weights close to pre-trained weights, and (2) ensuring the space of LoRA initialization is orthogonal to pre-trained knowledge. The former is what current methods strive to achieve, while the importance of the latter is not sufficiently recognized. We find that the space of LoRA initialization is the key to preserving pre-trained knowledge rather than the residual weights. Existing methods like MiLoRA propose making the LoRA initialization space orthogonal to pre-trained weights. However, MiLoRA utilizes the null space of pre-trained weights. Compared to pre-trained weights, the input activations of pre-trained knowledge take into account the parameters of all previous layers as well as the input data, while pre-trained weights only contain information from the current layer. Moreover, we find that the effective ranks of input activations are much smaller than those of pre-trained weights. Thus, the null space of activations is more accurate and contains less pre-trained knowledge information compared to that of weights. Based on these, we introduce LoRA-Null, our proposed method that initializes LoRA in the null space of activations. Experimental results show that LoRA-Null effectively preserves the pre-trained world knowledge of LLMs while achieving good fine-tuning performance, as evidenced by extensive experiments. Code is available at {https://github.com/HungerPWAY/LoRA-Null}.