MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning

TL;DR

MoSLD tackles the multi-task fine-tuning challenge for large language models by sharing a single general-feature LoRA projection A across all experts in each layer while keeping task-specific projections B, enabling cross-task knowledge transfer with far fewer parameters. A dropout strategy on A further stabilizes training and mitigates overfitting and optimization imbalance in the mixture-of-LoRAs setting. Evaluated on six commonsense benchmarks with a mixed data setup, MoSLD demonstrates robust gains in both single-task and multi-task regimes and exhibits strong out-of-domain generalization, while maintaining favorable computation efficiency compared with MoLA and other baselines. The approach contributes a practical, scalable solution for efficient, robust multi-task fine-tuning of LLMs by decoupling general and task-specific knowledge and regulating their updates through dropout. Overall, MoSLD advances parameter-efficient knowledge sharing across tasks and domains, reducing interference and forgetting while sustaining strong performance.

Abstract

Recently, LoRA has emerged as a crucial technique for fine-tuning large pre-trained models, yet its performance in multi-task learning scenarios often falls short. In contrast, the MoE architecture presents a natural solution to this issue. However, it introduces challenges such as mutual interference of data across multiple domains and knowledge forgetting of various tasks. Additionally, MoE significantly increases the number of parameters, posing a computational cost challenge. Therefore, in this paper, we propose MoSLD, a mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these challenges by sharing the upper projection matrix in LoRA among different experts, encouraging the model to learn general knowledge across tasks, while still allowing the lower projection matrix to focus on the unique features of each task. The application of dropout alleviates the imbalanced update of parameter matrix and mitigates parameter overfitting in LoRA. Extensive experiments demonstrate that our model exhibits excellent performance in both single-task and multi-task scenarios, with robust out-of-domain generalization capabilities.

Figures (7)

• Figure 1: The increase between mixture setting and single setting for FP-tuning and LoRA on four datasets. The vertical axis is Score (mixture)-Score (single).
• Figure 2: Overview of the share mechansim and dropout strategy in our MoSLD. Noted that the matrix A is shared among all experts in each layer.
• Figure 3: The overview of our proposed Mixture-of-Shared-LoRA with dropout strategy applied on $W_q$ and $W_v$.
• Figure 4: A comparision of performance for LoRA, MoLA, MoSL, and MoSLD on single and mixture settings for MMLU test set.
• Figure 5: Results of six datasets under different dropout ratios. Here, we are based on the mixture setting.