ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimation

TL;DR

ShapLoRA tackles the inefficiency of uniform LoRA rank allocation in large language models by introducing a Shapley-value-inspired sensitivity score, termed Shapley sensitivity, which captures rank importance under coalitional interactions. The method computes this score on a separate validation set and uses a prune-then-finetune two-stage workflow to allocate ranks before final training, improving both accuracy and efficiency over prior PEFT baselines. Empirical results across QA, math reasoning, instruction tuning, and NLP/NLG tasks show ShapLoRA consistently surpasses state-of-the-art LoRA variants with comparable tunable parameters, while maintaining training and inference efficiency. The approach enhances explainability of rank contributions and demonstrates robust performance across backbones and seeds, offering practical benefits for deploying parameter-efficient LLM fine-tuning.

Abstract

Low-rank adaption (LoRA) is a representative method in the field of parameter-efficient fine-tuning (PEFT), and is key to Democratizating the modern large language models (LLMs). The vanilla LoRA is implemented with uniform ranks, and the recent literature have found that properly allocating ranks on the LLM backbones results in performance boosts. However, the previous rank allocation methods have limitations since they rely on inexplanable and unreliable importance measures for the LoRA ranks. To address the above issues, we propose the ShapLoRA framework. Inspired by the explanable attribution measure Shapley Value, we combine the sensitivity-based measures with the idea of coalitions in the collaborative games among LoRA ranks, and propose a more explainable importance measure called Shapley sensitivity. In addition, we optimize the workflow of the existing works by: (a) calculating Shapley sensitivity on a separate validation set; (b) Setting up the allocating-retraining procedures for fair comparisons. We have conducted experiments on various challenging tasks, and the experimental results demonstrate that our ShapLoRA method can outperform the recent baselines with comparable tunable parameters.\footnote{Codes and fine-tuned models will be open-sourced to facilitate future research.

Figures (5)

• Figure 1: Schematic illustration of our ShapLoRA framework. (Left): ShapLoRA follows LoRA and AdaLoRA to update the weight matrix $W$ by fine-tuning the low-rank incremental matrix $\Delta W = PGQ$ with a SVD formation. And LoRA rank $i$ is represented by a tuple ($G_{i,i}$, $P_{:,i}$, $Q_{i, :}$) of parameters. (Right): ShapLoRA prunes the less important LoRA rank $k$ by fixing $G_{k,k}$ to zero. Here the importance score $\text{IPT(k)}$ for each LoRA rank $k$ is calculated as the average sensitivity score of the parameters in rank $k$. Different from previous works, $\text{IPT(k)}$ is calculated under a random subset of coalitions in which each LoRA rank is considered as a player in a collaborative game. A coalition is formulated by randomly masking some of the elements of $G$ to zeros.
• Figure 2: Performances under different tunable parameter budgets. The $x$-axis represents the number of tunable parameters, and the $y$-axis represents the performance score.
• Figure 3: Shapley sensitivity importance scores of LoRA ranks on the 8th, 16th, 24th, and 32nd layers of LLaMA-3 8B after finetuning on BoolQ.
• Figure 4: Shapley sensitivity importance scores of LoRA ranks on the 8th, 16th, 24th, and 32nd layers of LLaMA-3 8B after finetuning on PIQA.
• Figure 5: ShapLoRA's LoRA rank allocation results on the LLaMA-3 8B backbone after finetuning on BoolQ and PIQA.