Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, Fei Wu

TL;DR

This work targets the problem of merging multiple LoRAs without retraining, where parameter interference often degrades performance. It introduces Minimal Semantic Units (MSUs) and a LEGO-inspired LoRA-LEGO framework that disassembles LoRAs into MSUs, clusters them by similarity, and reconstructs a merged LoRA from cluster centroids with adjustable rank. The method includes dual scaling to address norm decay and variance expansion, enabling robust merging across heterogeneous ranks and tasks. Empirical results on multi-task and mixed-task benchmarks show LoRA-LEGO outperforms existing post-hoc composition approaches and even supports substantial LoRA pruning while preserving performance. The approach bridges model merging and ensemble strategies, offering a scalable, flexible mechanism for composing diverse LoRAs in practical LLM adaptation workflows.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

TL;DR

This work targets the problem of merging multiple LoRAs without retraining, where parameter interference often degrades performance. It introduces Minimal Semantic Units (MSUs) and a LEGO-inspired LoRA-LEGO framework that disassembles LoRAs into MSUs, clusters them by similarity, and reconstructs a merged LoRA from cluster centroids with adjustable rank. The method includes dual scaling to address norm decay and variance expansion, enabling robust merging across heterogeneous ranks and tasks. Empirical results on multi-task and mixed-task benchmarks show LoRA-LEGO outperforms existing post-hoc composition approaches and even supports substantial LoRA pruning while preserving performance. The approach bridges model merging and ensemble strategies, offering a scalable, flexible mechanism for composing diverse LoRAs in practical LLM adaptation workflows.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of . Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.
Paper Structure (34 sections, 2 theorems, 15 equations, 10 figures, 5 tables)

This paper contains 34 sections, 2 theorems, 15 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Low-Rank Adaptation
  4. Further modularization of LoRA
  5. Problem Formulation and Challenges
  6. Methodology
  7. LoRA-LEGO framework
  8. MSU Grouping.
  9. MSU Clustering.
  10. LoRA Reconstruction.
  11. Optimal scale of Merged LoRA
  12. Norm Decay After LoRA Merging.
  13. Variance Expansion with Increased LoRA Rank.
  14. Experiments
  15. Multi-task Learning
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let ${\bm{A}}_1 \in \mathbb{R}^{p \times r}$ and ${\bm{B}}_1 \in \mathbb{R}^{r \times p}$, and ${\bm{A}}_2 \in \mathbb{R}^{p \times k}$ and ${\bm{B}}_2 \in \mathbb{R}^{k \times p}$, where all elements of these matrices are independently and identically distributed according to the standard normal di

Figures (10)

  • Figure 1: Further Modularization of LoRA: a) Each LoRA can be further modularized into multiple Minimal Semantic Units (MSUs), each corresponding to a row in ${\bm{A}}$ matrix and a column in matrix ${\bm{B}}$, differentiated by distinct colors. b) The MSUs within a LoRA display permutation invariance, implying that any rearrangement of the MSUs does not affect the output generated by the LoRA. c) Multiple LoRAs exhibit Concatenation-Summation Equivalence, indicating that the summation of outputs from various LoRAs is equivalent to the output of a singular LoRA constructed by concatenating their MSUs.
  • Figure 1: Performance degradation after merging misaligned LoRAs. "Original" refers to the performance of the unaltered LoRA, while "Misaligned" indicates the performance after merging the LoRA with a randomly permuted version of itself.
  • Figure 2: Two sources of parameter interference in LoRA merging. The left part illustrates how parameter misalignment can lead to interference; the right part demonstrates that knowledge conflict in merged LoRA layers can also result in parameter interference.
  • Figure 2: Parameter interference due to knowledge conflict. "Tuning MSU" indicates the performance after tuning the added MSU for each task. "Avg MSU" denotes the performance achieved by directly merging these task-specific MSUs. "Concat MSU" represents the performance after concatenating these task-specific MSUs.
  • Figure 3: The LoRA-LEGO framework merges candidate LoRAs in a manner akin to playing with LEGO by: a) first disassembling LoRAs into multiple MSUs and grouping them into an MSU pool; b) performing MSU clustering to merge similar MSUs; c) reconstructing the merged LoRA from the centroid MSUs to form a cohesive LoRA.
  • ...and 5 more figures

Theorems & Definitions (4)

  • Definition 1
  • Theorem 3.1
  • Theorem D.1
  • proof