MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

TL;DR

MoS addresses the memory bottleneck of servicing numerous LoRA-based finetuned models by introducing a Mixture of Shards that combines inter-layer and intra-layer sharing with four near-cost-free differentiation strategies. Using a MoE-like routing mechanism to select and concatenate shards from global pools, MoS significantly increases parameter efficiency while preserving LoRA's linearity and low-cost model switching. Empirical results show about 8x reductions in trainable parameters with competitive performance across diverse instruction-following benchmarks, and ablations confirm the essential roles of subset selection, pair dissociation, vector sharding, and shard privatization. The approach provides a practical pathway to scalable, parameter-efficient fine-tuning in large language model deployment, with broad implications for future parameter-sharing methods.

Abstract

The rapid scaling of large language models necessitates more lightweight finetuning methods to reduce the explosive GPU memory overhead when numerous customized models are served simultaneously. Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents a promising solution. Empirically, our research into high-level sharing principles highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing. Guided by this finding, we propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism before sequentially concatenating them to low-rank matrices. Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effectively circumvents the drawbacks of peer parameter-sharing methods. Our empirical experiments demonstrate approximately 8x parameter savings in a standard LoRA setting. The ablation study confirms the significance of each component. Our insights into parameter sharing and MoS method may illuminate future developments of more parameter-efficient finetuning methods. The code is officially available at https://github.com/Forence1999/MoS.

Figures (2)

• Figure 1: Illustration of vanilla LoRA, pure sharing, and differentiated pure sharing methods. The rank of vanilla LoRA is denoted as $r$, while that of pure sharing is boosted to $rL$ through the aggregation of trainable parameters among $L$ Transformer blocks. In Figure \ref{['fig: Pure Sharing (Differentiated)']}, the colored squares between the low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ signify scalars and boolean values for random scaling and subset selection strategies, respectively.
• Figure 2: Illustration of MoS from the perspectives of inter-layer and intra-layer sharing. (a) Across layers, each layer retrieves shards from the same global pool, utilizing an independent index matrix and a MoE-like router "R". (b) Within each layer, the shard retrieval process is visualized in details with the number of shards per vector $l$, rank $r$, pool size as 2, 4, and n, respectively. The circled numbers denote the shard indices in the global pool, while the small italicized numbers indicate the shard positions within the low-rank matrix. Blue highlights shared components, whereas green indicates privatized ones.