Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

Jiawei Hao, Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Dan Zeng

Abstract

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by substantial memory demands, primarily due to the need to load numerous expert modules. While existing expert compression techniques like pruning or merging attempt to mitigate this, they often suffer from irreversible knowledge loss or high training overhead. In this paper, we propose a novel expert compression paradigm termed expert replacing, which replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs. We find that even a straightforward baseline of this paradigm yields promising performance. Building on this foundation, we introduce LightMoE, a framework that enhances the paradigm by introducing adaptive expert selection, hierarchical expert construction, and an annealed recovery strategy. Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio. Even under a more aggressive 50% compression rate, it outperforms existing methods and achieves average performance improvements of 5.6% across five diverse tasks. These findings demonstrate that LightMoE strikes a superior balance among memory efficiency, training efficiency, and model performance.

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

Abstract

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by substantial memory demands, primarily due to the need to load numerous expert modules. While existing expert compression techniques like pruning or merging attempt to mitigate this, they often suffer from irreversible knowledge loss or high training overhead. In this paper, we propose a novel expert compression paradigm termed expert replacing, which replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs. We find that even a straightforward baseline of this paradigm yields promising performance. Building on this foundation, we introduce LightMoE, a framework that enhances the paradigm by introducing adaptive expert selection, hierarchical expert construction, and an annealed recovery strategy. Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio. Even under a more aggressive 50% compression rate, it outperforms existing methods and achieves average performance improvements of 5.6% across five diverse tasks. These findings demonstrate that LightMoE strikes a superior balance among memory efficiency, training efficiency, and model performance.
Paper Structure (23 sections, 9 equations, 8 figures, 9 tables)

This paper contains 23 sections, 9 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Mixture-of-Experts LLMs
  4. MoE Compression Methods
  5. Low-Rank Adaptation
  6. Method
  7. Adaptive Expert Selection
  8. Hierarchical Expert Construction
  9. Annealed Expert Replacement
  10. Experiments
  11. Experiment Setup
  12. Main Results
  13. Ablation Study
  14. Conclusion
  15. Supplemental Experimental Results
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Performance comparison across different compression ratios on the Math task. While the directly replacing strategy performs comparably to MC-SMoE, both suffer from significant performance degradation, particularly at higher compression ratios
  • Figure 2: Overview of the proposed LightMoE framework. (Left) A standard MoE layer. (Right) The LightMoE workflow, comprising three key steps: (1) scoring experts and selecting those with lower scores as compression candidates, (2) grouping the selected candidates, and (3) replacing each group with a shared base augmented with lightweight, expert-specific adaptation parameters.
  • Figure 3: Analysis of expert importance in OLMoE-1B-7B-SFT.
  • Figure 4: Comparison of different end ratios for LightMoE on the Math task. Directly replacing (blue points) is consistently sub-optimal, which shows the effectiveness of our annealed expert replacement strategy (red points).
  • Figure 5: Comparison of different ranks at different compression ratios on the Math task.
  • ...and 3 more figures