Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

Chongjie Si, Zhiyi Shi, Yadao Wang, Xiaokang Yang, Susanto Rahardja, Wei Shen

TL;DR

MAP recasts weight matrices as flattened vectors and decouples adaptation into direction and magnitude using two learnable scalars, providing a principled, lightweight enhancement to parameter-efficient fine-tuning. The vector-based (and equivalently matrix-based) formulations enable clean, interpretable control of the base weights and their updates, and MAP can be integrated with existing methods such as LoRA to form LoMAP. Across commonsense reasoning, natural language understanding, and generation tasks, MAP-derived configurations consistently outperform strong baselines while maintaining minimal parameter overhead and similar training costs. The work introduces a practical framework with broad compatibility, suggesting MAP as a default design choice for future PEFT methodologies.

Abstract

The rapid development of large language models has revolutionized natural language processing, but their fine-tuning remains computationally expensive, hindering broad deployment. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have emerged as solutions. Recent work like DoRA attempts to further decompose weight adaptation into direction and magnitude components. However, existing formulations often define direction heuristically at the column level, lacking a principled geometric foundation. In this paper, we propose MAP, a novel framework that reformulates weight matrices as high-dimensional vectors and decouples their adaptation into direction and magnitude in a rigorous manner. MAP normalizes the pre-trained weights, learns a directional update, and introduces two scalar coefficients to independently scale the magnitude of the base and update vectors. This design enables more interpretable and flexible adaptation, and can be seamlessly integrated into existing PEFT methods. Extensive experiments show that MAP significantly improves performance when coupling with existing methods, offering a simple yet powerful enhancement to existing PEFT methods. Given the universality and simplicity of MAP, we hope it can serve as a default setting for designing future PEFT methods.

MAP: Revisiting Weight Decomposition for Low-Rank Adaptation

TL;DR

MAP recasts weight matrices as flattened vectors and decouples adaptation into direction and magnitude using two learnable scalars, providing a principled, lightweight enhancement to parameter-efficient fine-tuning. The vector-based (and equivalently matrix-based) formulations enable clean, interpretable control of the base weights and their updates, and MAP can be integrated with existing methods such as LoRA to form LoMAP. Across commonsense reasoning, natural language understanding, and generation tasks, MAP-derived configurations consistently outperform strong baselines while maintaining minimal parameter overhead and similar training costs. The work introduces a practical framework with broad compatibility, suggesting MAP as a default design choice for future PEFT methodologies.

Abstract

The rapid development of large language models has revolutionized natural language processing, but their fine-tuning remains computationally expensive, hindering broad deployment. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have emerged as solutions. Recent work like DoRA attempts to further decompose weight adaptation into direction and magnitude components. However, existing formulations often define direction heuristically at the column level, lacking a principled geometric foundation. In this paper, we propose MAP, a novel framework that reformulates weight matrices as high-dimensional vectors and decouples their adaptation into direction and magnitude in a rigorous manner. MAP normalizes the pre-trained weights, learns a directional update, and introduces two scalar coefficients to independently scale the magnitude of the base and update vectors. This design enables more interpretable and flexible adaptation, and can be seamlessly integrated into existing PEFT methods. Extensive experiments show that MAP significantly improves performance when coupling with existing methods, offering a simple yet powerful enhancement to existing PEFT methods. Given the universality and simplicity of MAP, we hope it can serve as a default setting for designing future PEFT methods.

Paper Structure

This paper contains 26 sections, 5 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter Efficient Fine-tuning
  4. Low-rank Adaptation
  5. Advancement in Low-rank Adaptation
  6. Weigh Decomposed Low-rank Adaptation
  7. The Proposed Framework: MAP
  8. Vector-Based Formulation
  9. Matrix-Based Formulation
  10. Experiments
  11. Commonsense Reasoning
  12. Task, Model, and Baselines
  13. Experimental Results
  14. Natural Language Understanding
  15. Task, Model, and Baselines
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Comparison of LoRA, DoRA, and our proposed MAP framework. DoRA normalizes the sum $\mathbf{W} + \mathbf{A}\mathbf{B}$ column-wise and rescales each column using a trainable vector. In contrast, MAP normalizes both $\mathbf{W}$ and $\mathbf{A}\mathbf{B}$ using their Frobenius norms, and applies two learnable scalar coefficients $\alpha$ and $\beta$ to decouple and modulate their magnitudes. MAP provides a more principled and compact decoupling strategy in the vector space.
  • Figure 2: Comparison of generated images from LoRA and LoMAP on the subject-driven generation task. It is evident that LoMAP consistently produces images that better reflect both the input subjects and the intended prompts compared to standard LoRA.