MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition

TL;DR

A two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost and can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks is introduced.

Abstract

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$\times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks. The code will be available at \url{https://github.com/xiaochengsky/MoEI-2.git}

Figures (5)

• Figure 1: The three-stage pipeline of MoE-I$^2$. The first stage (left) represents Inter-Expert Pruning, where MoE-I$^2$ conducts the Layer Importance Analysis on the target MoE model. By using a predefined overall pruning rate, it determines varying pruning ratios of different layers. Subsequently, the unimportant experts in MoE are determined by Layer-wise Genetic Search and Block-wise KT-Reception Field. The MoE is pruned accordingly. The second stage (middle) represents Intra-Expert Analysis. Similarly, MoE-I$^2$ automatically performs Expert Importance Analysis on the pruned model and using a predefined overall decomposition rate, applies varying ranks and low-rank decomposition to different experts, resulting in a final compressed model. The third stage (right) shows that we fine-tuned the compressed MoE model to recover performance.
• Figure 2: Importance analysis of Mixtral-8$\times$7B (left), Qwen1.5-MoE-A2.7B (middle), and DeepSeek-V2-Lite (right) models. A larger loss indicates greater importance. For Mixtral-8$\times$7B and Qwen1.5-MoE-A2.7B, the importance of the different layers is relatively consistent, but for DeepSeek-V2-Lite, the importance increases as one approaches the output layer.
• Figure 3: The process of KT-Receptive Filed in Mixtral-8$\times$7B for satisfied 25% pruning ratio. In this case, the number of candidate combinations per layer is $K=3$, and the number of layers per block is $T = 3$. For each layer, we select $K$ optimal candidates using the Layer-wise Genetic Search (top-left). Within a consecutive sequence of $T$ layers, we employ the Block-wise KT-Reception Field to identify the best-performing combination within that block ($T$ layers).
• Figure 4: The left and right figures represent the loss convergence for each layer of Qwen1.5-MoE-A2.7B and DeepSeek-V2-Lite during the Genetic Search process, respectively. As shown in the figures, after 50 iterations, nearly all layers have converged.
• Figure 5: The impact of $K$ and $T$ on the performance of models Mixtral-8$\times$7B, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite.