OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning

TL;DR

OPLoRA addresses catastrophic forgetting in parameter-efficient fine-tuning by constraining LoRA updates to the orthogonal complement of the dominant pre-trained subspaces. It achieves this with double-sided projections, yielding $\Delta W = P_L B A P_R$ that preserves the top-$k$ singular triples of the frozen weight matrix, guaranteed by Propositions 1 and 2. The approach introduces $\rho_k$ to quantify subspace interference and demonstrates, across commonsense, mathematical reasoning, and code-generation tasks on LLaMA-2-7B and Qwen2.5-7B, that forgetting is significantly reduced while maintaining competitive task performance. This subspace-aware PEFT method offers principled knowledge preservation with practical impact for robust deployment of large language models during continual and task-specific fine-tuning.

Abstract

Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-$k$ singular subspace using projections $P_L = I - U_k U_k^\top$ and $P_R = I - V_k V_k^\top$. We prove that this construction exactly preserves the top-$k$ singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce $ρ_k$, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.

Figures (2)

• Figure 1: Comparison of standard LoRA (left) and our proposed OPLoRA (right). The pre-trained weight matrix $W$ is decomposed via singular value decomposition (SVD) as $W = U \Sigma V^\top$. During fine-tuning, the standard LoRA update $BA$ is sandwiched between two frozen projection matrices, $P_L = (I - U_k U_k^\top)$ and $P_R = (I - V_k V_k^\top)$. This structure constrains the low-rank adaptation to lie in the orthogonal complement of the pre-trained dominant subspace, mitigating interference with essential model knowledge and reducing catastrophic forgetting.
• Figure 2: Quantifying and Mitigating Forgetting in LoRA-based Fine-Tuning. (a) Subspace alignment $\rho_k$ between the learned LoRA update and the top-$k$ singular directions of the pre-trained weight matrix $W_0$ in the LLaMA-2 7B layers.0.self.attn.q_proj. A higher $\rho_k$ indicates stronger interference with pre-trained knowledge. OPLoRA-128 achieves the lowest alignment across all $k$, indicating better preservation of prior knowledge. Note that $\rho_k$ is not exactly zero for OPLoRA-128 due to the use of a low-rank SVD approximation (svd_lowrank) for computational efficiency. (b) Average forgetting scores evaluated on three held-out commonsense reasoning tasks after fine-tuning. Higher scores indicate better retention of pre-trained knowledge. Methods are sorted by their corresponding subspace alignment value $\rho_{16}$ (shown in (a)). A clear negative correlation is observed: lower $\rho_{16}$ aligns with better forgetting robustness.