The Appeal and Reality of Recycling LoRAs with Adaptive Merging

TL;DR

This work tackles the practical question of recycling LoRAs from real-world public repositories using adaptive merging. It introduces a unified framework to evaluate design choices and conducts a large-scale study over a pool of $N=958$ LoRAs on $62$ downstream tasks, comparing adaptive merging to a base model and to target-task LoRAs. The findings show that adaptive merging offers gains over prompting but generally underperforms training a dedicated target-task LoRA; including the target-task LoRA in the pool often minimizes differences between selection strategies, and randomly initialized LoRAs can perform nearly as well as carefully chosen ones, suggesting a regularization effect rather than knowledge transfer. The work emphasizes the need for realistic, in-the-wild evaluation and better public LoRA infrastructure, and it releases checkpoints and code to support replication and further research.

Abstract

The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These methods typically include some way of selecting LoRAs from a pool and tune merging coefficients based on a task-specific dataset. While adaptive merging methods have demonstrated improvements in some settings, no past work has attempted to recycle LoRAs found "in the wild" on model repositories like the Hugging Face Hub. To address this gap, we consider recycling from a pool of nearly 1,000 user-contributed LoRAs trained from the Llama 3.1 8B-Instruct language model. Our empirical study includes a range of adaptive and non-adaptive merging methods in addition to a new method designed via a wide search over the methodological design space. We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA on the same data used to set merging coefficients. We additionally find not only that the specific choice of LoRAs to merge has little importance, but that using LoRAs with randomly initialized parameter values yields similar performance. This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect, rather than by enabling positive cross-task transfer. To better understand why past work has proven successful, we confirm that positive transfer is indeed possible when there are highly relevant LoRAs in the pool. We release the model checkpoints and code online.

Figures (14)

• Figure 1: Rescaled LoRA performance across 62 downstream tasks, categorized into five buckets based on their relative % improvement over prompting or LoRA baselines. LoRAs are sorted along the x-axis by their performance.
• Figure 2: Avg. % acc. improvement from baselines across all 62 tasks, without and with the target-task LoRA in the merging pool. TSV, TIES, and Simple Averaging are non-adaptive methods, while AdaMerging, $\pi$-Tuning, LoraHub, and Ours are adaptive methods.
• Figure 3: Adding the target-task LoRA significantly reduces the impact of LoRA selection method. The average LoRA baseline performance across tasks is shown as a gray line.
• Figure 4: Distribution of merging coefficients across 20 evaluation-based LoRAs for 62 downstream tasks, a) without and b) with the target-task LoRA in the merging pool.
• Figure 5: (a) In-house LoRAs outperform recycled LoRAs. (b) Excluding top-ranked in-house LoRAs degrades merging performance, converging to the randomly reinitialized LoRA performance.