LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

TL;DR

LoRA-Mini is proposed, an optimized adaptation of LoRA that improves parameter efficiency by splitting low-rank matrices into four parts, with only the two inner matrices being trainable.

Abstract

The rapid advancements in large language models (LLMs) have revolutionized natural language processing, creating an increased need for efficient, task-specific fine-tuning methods. Traditional fine-tuning of LLMs involves updating a large number of parameters, which is computationally expensive and memory-intensive. Low-Rank Adaptation (LoRA) has emerged as a promising solution, enabling parameter-efficient fine-tuning by reducing the number of trainable parameters. However, while LoRA reduces the number of trainable parameters, LoRA modules still create significant storage challenges. We propose LoRA-Mini, an optimized adaptation of LoRA that improves parameter efficiency by splitting low-rank matrices into four parts, with only the two inner matrices being trainable. This approach achieves upto a 20x reduction compared to standard LoRA in the number of trainable parameters while preserving performance levels comparable to standard LoRA, addressing both computational and storage efficiency in LLM fine-tuning.

Figures (2)

• Figure 1: A visual comparison of the LoRA and LoRA-Mini techniques.
• Figure 2: Accuracy comparison across different values of $r$ for four tasks using LoRA-Mini applied to the Attention and Dense layers of RoBERTa-Base. The graph reports average of all combinations of a and b in that rank.