Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen

TL;DR

The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved.

Abstract

Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

TL;DR

The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved.

Abstract

Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.
Paper Structure (33 sections, 5 equations, 2 figures, 14 tables)

This paper contains 33 sections, 5 equations, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Singular Value Decomposition
  4. Low-Rank Adaptation
  5. Methodology
  6. Experiments
  7. Experiments on Large Language Model
  8. Models and Datasets
  9. Implementation Details
  10. Results
  11. Experiments on Vision-Language Model
  12. Models and Datasets
  13. Implementation Details
  14. Results
  15. Understanding MiLoRA
  16. ...and 18 more sections

Figures (2)

  • Figure 1: (a) MiLoRA method splits the pretrained linear weight matrix into two parts, the minor singular component (${\bm{U}}_m, {\bm{V}}_m, {\bm \Sigma}_m$) is used to initialize the low-rank matrices ${\bm{A}}_m$ and ${\bm{B}}_m$ in the LoRA framework. (b) During finetuning, only the low-rank matrices ${\bm{A}}_m$ and ${\bm{B}}_m$ are updated while the principal matrix ${\bm{W}}_p$ is frozen, which is similar to the practice of LoRA method.
  • Figure 2: Left: The subspace similarity averaged by all modules in a layer. Right: The subspace similarity averaged by all layers for a specific module type.