Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

Junzhou Xu, Boyu Diao

TL;DR

This work tackles the challenge of parameter redundancy and degraded multi-task performance in LoRA-MoE fine-tuning by introducing LoRA-SMoE, a sensitivity-driven expert allocation method. It estimates parameter block sensitivity via gradient-based metrics on sampled data and adaptively assigns experts under a fixed budget, preserving LoRA's memory footprint. Across eight benchmarks with a 36-layer Qwen2.5-3B-Instruct base, the method—especially the LoRA-SMoE-S variant that treats attention and MLP modules separately—achieves superior performance with fewer trainable parameters and reduced overfitting. The findings reveal that higher-layer attention and certain lower/upper MLP blocks are most sensitive to downstream tasks, offering practical guidance for efficient resource-aware multi-task fine-tuning.

Abstract

As deep learning models expand, the pre-training-fine-tuning paradigm has become the standard approach for handling various downstream tasks. However, shared parameters can lead to diminished performance when dealing with complex datasets involving multiple tasks. While introducing Mixture-of-Experts (MoE) methods has alleviated this issue to some extent, it also significantly increases the number of parameters required for fine-tuning and training time, introducing greater parameter redundancy. To address these challenges, we propose a method for allocating expert numbers based on parameter sensitivity LoRA-SMoE (A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning). This method rapidly assesses the sensitivity of different tasks to parameters by sampling a small amount of data and using gradient information. It then adaptively allocates expert numbers within a given budget. The process maintains comparable memory consumption to LoRA (Low-Rank Adaptation) while ensuring an efficient and resource-friendly fine-tuning procedure. Experimental results demonstrate that compared to SOTA fine-tuning methods, our LoRA-SMoE approach can enhance model performance while reducing the number of trainable parameters. This significantly improves model performance in resource-constrained environments. Additionally, due to its efficient parameter sensitivity evaluation mechanism, LoRA-SMoE requires minimal computational overhead to optimize expert allocation, making it particularly suitable for scenarios with limited computational resources. All the code in this study will be made publicly available following the acceptance of the paper for publication. Source code is at https://github.com/EMLS-ICTCAS/LoRA-SMoE

A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning

TL;DR

This work tackles the challenge of parameter redundancy and degraded multi-task performance in LoRA-MoE fine-tuning by introducing LoRA-SMoE, a sensitivity-driven expert allocation method. It estimates parameter block sensitivity via gradient-based metrics on sampled data and adaptively assigns experts under a fixed budget, preserving LoRA's memory footprint. Across eight benchmarks with a 36-layer Qwen2.5-3B-Instruct base, the method—especially the LoRA-SMoE-S variant that treats attention and MLP modules separately—achieves superior performance with fewer trainable parameters and reduced overfitting. The findings reveal that higher-layer attention and certain lower/upper MLP blocks are most sensitive to downstream tasks, offering practical guidance for efficient resource-aware multi-task fine-tuning.

Abstract

As deep learning models expand, the pre-training-fine-tuning paradigm has become the standard approach for handling various downstream tasks. However, shared parameters can lead to diminished performance when dealing with complex datasets involving multiple tasks. While introducing Mixture-of-Experts (MoE) methods has alleviated this issue to some extent, it also significantly increases the number of parameters required for fine-tuning and training time, introducing greater parameter redundancy. To address these challenges, we propose a method for allocating expert numbers based on parameter sensitivity LoRA-SMoE (A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning). This method rapidly assesses the sensitivity of different tasks to parameters by sampling a small amount of data and using gradient information. It then adaptively allocates expert numbers within a given budget. The process maintains comparable memory consumption to LoRA (Low-Rank Adaptation) while ensuring an efficient and resource-friendly fine-tuning procedure. Experimental results demonstrate that compared to SOTA fine-tuning methods, our LoRA-SMoE approach can enhance model performance while reducing the number of trainable parameters. This significantly improves model performance in resource-constrained environments. Additionally, due to its efficient parameter sensitivity evaluation mechanism, LoRA-SMoE requires minimal computational overhead to optimize expert allocation, making it particularly suitable for scenarios with limited computational resources. All the code in this study will be made publicly available following the acceptance of the paper for publication. Source code is at https://github.com/EMLS-ICTCAS/LoRA-SMoE
Paper Structure (20 sections, 3 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 20 sections, 3 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. LoRA
  4. MoE
  5. Sensitivity
  6. Motivation
  7. LoRA-SMoE
  8. Get Parameter Sensitivity
  9. Adaptive Experts Allocation
  10. Unified Selection of Sensitive Parameters (LoRA-SMoE-U)
  11. Separate Selection of Parameters from Self-Attention and MLP Layers (LoRA-SMoE-S)
  12. Independent Allocation for Each Matrix (LoRA-SMoE-I)
  13. Fine-tuning
  14. Experiments
  15. Tasks and Datasets
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Parameter blocks sensitivity of Qwen2.5-3B-Instruct on eight datasets, the darker the color, the higher the sensitivity.
  • Figure 2: LoRA-SMoE Framework Overview(steps A and B show our method for one layer's parameter blocks)
  • Figure 3: Parameter blocks sensitivity obtained with different sampling sizes