Mitigating Forgetting in Low Rank Adaptation

TL;DR

LaLoRA introduces a lightweight, curvature-aware regularizer for LoRA-based fine-tuning by applying a Laplace approximation to the trainable LoRA adapters. By estimating parameter importance from surrogate source data and penalizing changes in high-curvature directions, it preserves pre-trained knowledge while enabling target-domain learning. The method yields a superior learning-forgetting Pareto frontier compared to baselines across multiple tasks and models, and demonstrates robustness to hyperparameters and surrogate data choices. Its modular design supports several curvature schemes (Diag, B-KFAC, B-Tri-KFAC) and remains practical with minimal source data. This approach provides a principled path toward more robust, uncertainty-aware parameter-efficient fine-tuning for large language models.

Abstract

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), enable fast specialization of large pre-trained models to different downstream applications. However, this process often leads to catastrophic forgetting of the model's prior domain knowledge. We address this issue with LaLoRA, a weight-space regularization technique that applies a Laplace approximation to Low-Rank Adaptation. Our approach estimates the model's confidence in each parameter and constrains updates in high-curvature directions, preserving prior knowledge while enabling efficient target-domain learning. By applying the Laplace approximation only to the LoRA weights, the method remains lightweight. We evaluate LaLoRA by fine-tuning a Llama model for mathematical reasoning and demonstrate an improved learning-forgetting trade-off, which can be directly controlled via the method's regularization strength. We further explore different loss landscape curvature approximations for estimating parameter confidence, analyze the effect of the data used for the Laplace approximation, and study robustness across hyperparameters.

Figures (27)

• Figure 1: Our Laplace regularizer LaLoRA maintains learning while limiting source domain forgetting. The left panel illustrates the standard LoRA setup, where the pre-trained weights${\bm{W}}_0$ remain frozen and only the LoRA adapters${\bm{A}}$ and ${\bm{B}}$ are trained. The center panel visualizes the Laplace approximation to the LoRA weights, supporting different curvature approximations to identify parameters critical for source performance. The right panel presents the LaLoRA algorithm in two parts: (I) uncertainty quantification of the trainable weights via the LaplaceApprox and (II)Laplace-regularized fine-tuning on the target domain that mitigates source forgetting.
• Figure 2: Mitigating source domain forgetting: (a) Forgetting and learning dynamics over the course of fine-tuning on math reasoning data and (b) the resulting trade-off.
• Figure 3: Analysis of LaLoRA's update pattern and data dependency. (a) Unlike the baseline, LaLoRA limits updates to flexible (low-precision) weights. (b) Using a more comprehensive set of surrogate datasets for the Laplace approximation leads to less source-domain forgetting.
• Figure 4: Two batches per source dataset optimizes the learning--forgetting trade-off. We analyze the impact of using one, two, or three batches ($y$-axis) per surrogate dataset for the Laplace approximation. Performance is measured as benefit over baseline (orange is better) for both target-domain learning (left) and source-domain accuracy (right) across three regularization strengths $\lambda$ ($x$-axis). The mean benefit subplots summarize the results across $\lambda$ and three seeds.
• Figure 5: Illustration of LoRA and its input and outputs.${\bm{A}}$ and ${\bm{B}}$ denote the adapter weigths at initialization, ${\bm{a}}$ activations, ${\bm{s}}$ pre-activations and ${\bm{x}}$ the input to $\Delta {\bm{W}}$.