Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixture of Routers

Jia-Chen Zhang, Yu-Jie Xiong, Xi-He Qiu, Chun-Ming Xia, Fei Dai, Zheng Zhou

TL;DR

This work targets the fragility of single-router routing in MoE-based, parameter-efficient fine-tuning of large language models. It introduces Mixture of Routers (MoR), a plug-in architecture with multiple sub-routers and a learnable main router, augmented by a load-balancing loss to ensure even expert usage. Across six NLP and commonsense benchmarks, MoR yields average improvements around 1% over strong MoE-LoRA baselines in both direct and instruction-based transfer fine-tuning, with notable gains on tasks like CommonsenseQA and OpenbookQA. Analyses show that using two sub-routers provides near-optimal performance with modest overhead, and MoR enhances routing reliability and load balancing, enabling broader, efficient deployment of MoE-based PEFT methods.

Abstract

Supervised fine-tuning (SFT) is a milestone in aligning large language models with human instructions and adapting them to downstream tasks. In particular, Low-Rank Adaptation (LoRA) has gained widespread attention due to its parameter efficiency. However, its impact on improving the performance of large models remains limited. Recent studies suggest that combining LoRA with Mixture-of-Experts (MoE) can significantly enhance fine-tuning performance. MoE adapts to the diversity and complexity of datasets by dynamically selecting the most suitable experts, thereby improving task accuracy and efficiency. Despite impressive results, recent studies reveal issues in the MoE routing mechanism, such as incorrect assignments and imbalanced expert allocation. Inspired by the principles of Redundancy and Fault Tolerance Theory. We innovatively integrate the concept of Mixture of Experts into the routing mechanism and propose an efficient fine-tuning method called Mixture of Routers (MoR). It employs multiple sub-routers for joint selection and uses a learnable main router to determine the weights of the sub-routers. The results show that MoR outperforms baseline models on most tasks, achieving an average performance improvement of 1%. MoR can serve as a plug-and-play, parameter-efficient fine-tuning method suitable for a wide range of applications. Our code is available here: https://anonymous.4open.science/r/MoR-DFC6.

Mixture of Routers

TL;DR

This work targets the fragility of single-router routing in MoE-based, parameter-efficient fine-tuning of large language models. It introduces Mixture of Routers (MoR), a plug-in architecture with multiple sub-routers and a learnable main router, augmented by a load-balancing loss to ensure even expert usage. Across six NLP and commonsense benchmarks, MoR yields average improvements around 1% over strong MoE-LoRA baselines in both direct and instruction-based transfer fine-tuning, with notable gains on tasks like CommonsenseQA and OpenbookQA. Analyses show that using two sub-routers provides near-optimal performance with modest overhead, and MoR enhances routing reliability and load balancing, enabling broader, efficient deployment of MoE-based PEFT methods.

Abstract

Supervised fine-tuning (SFT) is a milestone in aligning large language models with human instructions and adapting them to downstream tasks. In particular, Low-Rank Adaptation (LoRA) has gained widespread attention due to its parameter efficiency. However, its impact on improving the performance of large models remains limited. Recent studies suggest that combining LoRA with Mixture-of-Experts (MoE) can significantly enhance fine-tuning performance. MoE adapts to the diversity and complexity of datasets by dynamically selecting the most suitable experts, thereby improving task accuracy and efficiency. Despite impressive results, recent studies reveal issues in the MoE routing mechanism, such as incorrect assignments and imbalanced expert allocation. Inspired by the principles of Redundancy and Fault Tolerance Theory. We innovatively integrate the concept of Mixture of Experts into the routing mechanism and propose an efficient fine-tuning method called Mixture of Routers (MoR). It employs multiple sub-routers for joint selection and uses a learnable main router to determine the weights of the sub-routers. The results show that MoR outperforms baseline models on most tasks, achieving an average performance improvement of 1%. MoR can serve as a plug-and-play, parameter-efficient fine-tuning method suitable for a wide range of applications. Our code is available here: https://anonymous.4open.science/r/MoR-DFC6.

Paper Structure

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Motivation
  5. Mixture of Routers
  6. Experiments
  7. Experimental settings
  8. Main results
  9. Direct fine-tuning
  10. Transfer fine-tuning
  11. Analysis
  12. Router number analysis
  13. Efficiency analysis
  14. Expert allocation analysis
  15. Conclusion

Figures (2)

  • Figure 1: Here is a schematic illustration of MoR. On the left is a schematic diagram of the integration of LoRA and MoE, incorporating LoRA as an expert into the MoE model. On the right is the MoR plugin we propose, which can flexibly replace the router layer in MoE-style models.
  • Figure 2: Visualizing the effect of incorporating the MoR plugin into the model through expert activation patterns with 8 experts.