Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

Hamid Nasiri, Peter Garraghan

TL;DR

EDoRA addresses the scalability and learning-pattern limitations of existing PEFT methods by decomposing pre-trained weights into magnitude and directional components and reparameterizing updates using a frozen low-rank pair with a small trainable interposed matrix. Leveraging SVD-based initialization, EDoRA aligns adaptation with the most informative directions of the pre-trained subspace, enabling substantial parameter reduction without sacrificing performance. Empirical results on the GLUE benchmark show EDoRA achieving competitive or superior accuracy with up to 30x fewer trainable parameters than LoRA and DoRA, driven by its magnitude-direction decomposition and stable initialization. This approach offers a practical path for memory-constrained deployment of LLM adapters across diverse tasks while preserving or enhancing learning capacity.

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters. However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning. To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components. By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity. Experimental results on the GLUE benchmark demonstrate that EDoRA achieves competitive or superior performance compared to state-of-the-art methods, such as LoRA and DoRA, with up to 30x fewer trainable parameters. This makes EDoRA a highly efficient solution for adapting LLMs to diverse tasks under memory-constrained settings. Code is available at https://github.com/Hamid-Nasiri/EDoRA .

EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

TL;DR

EDoRA addresses the scalability and learning-pattern limitations of existing PEFT methods by decomposing pre-trained weights into magnitude and directional components and reparameterizing updates using a frozen low-rank pair with a small trainable interposed matrix. Leveraging SVD-based initialization, EDoRA aligns adaptation with the most informative directions of the pre-trained subspace, enabling substantial parameter reduction without sacrificing performance. Empirical results on the GLUE benchmark show EDoRA achieving competitive or superior accuracy with up to 30x fewer trainable parameters than LoRA and DoRA, driven by its magnitude-direction decomposition and stable initialization. This approach offers a practical path for memory-constrained deployment of LLM adapters across diverse tasks while preserving or enhancing learning capacity.

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters. However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning. To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components. By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity. Experimental results on the GLUE benchmark demonstrate that EDoRA achieves competitive or superior performance compared to state-of-the-art methods, such as LoRA and DoRA, with up to 30x fewer trainable parameters. This makes EDoRA a highly efficient solution for adapting LLMs to diverse tasks under memory-constrained settings. Code is available at https://github.com/Hamid-Nasiri/EDoRA .
Paper Structure (13 sections, 8 equations, 4 figures, 4 tables)

This paper contains 13 sections, 8 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Additive Methods
  4. Selective Methods
  5. Reparametrization-based Methods
  6. Proposed Method
  7. Parameter Efficiency Analysis
  8. Experiments
  9. Experimental Setup
  10. GLUE Benchmark
  11. Ablation Study
  12. Conclusion
  13. Hyperparameter Configurations

Figures (4)

  • Figure 1: An overview of EDoRA
  • Figure 2: Relationship between the number of trainable parameters and average performance.
  • Figure 3: Impact of rank on average performance across different methods.
  • Figure 4: The impact of rank on average performance across different methods. The number of trainable parameters is shown above the bar plots for reference. At each rank, EDoRA is compared with LoRA, DoRA and LoRA-XS. For clarity, EDoRA’s average performance is displayed alongside the best-performing method among the three.