DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation

TL;DR

DeLoRA addresses robustness weaknesses in LoRA and expressivity limits in bound-based finetuning by normalizing low-rank updates and introducing a learnable boundary that decouples angular learning from adaptation strength. Derived from both LoRA and ETHER perspectives, DeLoRA uses a diagonal normalization matrix $\Xi$ and a layer-specific bound $\lambda$ to bound updates via $\Delta W = \frac{\lambda\|\bar{W}\|}{r} B\Xi A$, with optional weight-norm scaling to align updates with pretrained magnitudes. Empirical results across subject-driven image generation, semantic map-to-image, NLP understanding, and instruction tuning show DeLoRA matching or surpassing LoRA and ETHER baselines while exhibiting enhanced learning-rate robustness and resilience to prolonged training. The work demonstrates that decoupled, norm-bound updates enable high-rank, expressive finetuning without sacrificing stability, offering practical benefits for personalization, merging, and cross-domain adaptation in large pretrained models. Overall, DeLoRA provides a robust, flexible PEFT framework that combines low-rank efficiency with bounded updates to improve reliability in real-world finetuning scenarios.

Abstract

Parameter-Efficient FineTuning (PEFT) methods have recently gained significant popularity thanks to the widespread availability of large-scale pretrained models. These methods allow for quick adaptation to downstream tasks with minimal computational cost. However, popular finetuning methods such as LoRA exhibit limited robustness when it comes to hyperparameter choices or extended training regimes, preventing optimal out-of-the-box performance. In contrast, bounded approaches, such as ETHER, provide greater robustness but are limited to extremely low-rank adaptations and fixed-strength transformations, reducing their adaptation expressive power. In this work, we propose Decoupled Low-rank Adaptation (DeLoRA), a novel finetuning method that normalizes and scales learnable low-rank matrices. By bounding the distance of the transformation, DeLoRA effectively decouples the angular learning from the adaptation strength, enhancing robustness without compromising performance. Through evaluations on subject-driven image generation, natural language understanding, and instruction tuning, we show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness. Code is available at https://github.com/ExplainableML/DeLoRA.

Figures (8)

• Figure 1: Visualizations (Left) of the original LoRA hu2022lora and (Right) of our proposed method DeLoRA. In addition to the low-rank matrices $B,A$, we introduce a normalization $\Xi$ and a scaling factor $\lambda$, which effectively decouple the angular learning from the adaptation strength.
• Figure 2: Learning rate robustness plots in Subject-driven generation task in terms of DINO scores (Left) and Euclidean distance between a finetuned vs pretrained projection layer weights (Right). Learning rates used for robustness evaluation were derived by multiplying the base learning rate in a range of factors.
• Figure 3: (Left) Euclidean Distance of finetuned weights to pretrained weights as a function of the number of training steps. (Right) Qualitative examples show that LoRA exhibits significant artifacts earlier in the process compared to DeLoRA, which maintains better image quality.
• Figure 4: Average column norms of parameters in the attention modules of Stable Diffusion's Unet
• Figure 5: Robustness analysis between DoRA with and without magnitude updates, with respect to learning rate changes from the optimal learning rate.