Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density

Zhendong Mi, Yixiao Chen, Pu Zhao, Xiaodong Yu, Hao Wang, Yanzhi Wang, Shaoyi Huang

TL;DR

RFID-MoE tackles the memory bottleneck of mixture-of-experts LLMs by adaptively distributing compression ranks across expert groups according to both routing frequency and information density, instead of assuming uniform importance. It introduces a fused importance metric and a parameter-efficient residual reconstruction via a shared sparse projection to recover information discarded by low-rank approximations. Empirical results across diverse MoE models show RFID-MoE consistently surpasses state-of-the-art baselines in perplexity and zero-shot reasoning at high compression ratios, including notable gains on PTB and HellaSwag. The approach enables practical deployment of massive MoE models on resource-limited hardware while preserving core capabilities and fast convergence during compression.

Abstract

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and D2-MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%.

Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density

TL;DR

RFID-MoE tackles the memory bottleneck of mixture-of-experts LLMs by adaptively distributing compression ranks across expert groups according to both routing frequency and information density, instead of assuming uniform importance. It introduces a fused importance metric and a parameter-efficient residual reconstruction via a shared sparse projection to recover information discarded by low-rank approximations. Empirical results across diverse MoE models show RFID-MoE consistently surpasses state-of-the-art baselines in perplexity and zero-shot reasoning at high compression ratios, including notable gains on PTB and HellaSwag. The approach enables practical deployment of massive MoE models on resource-limited hardware while preserving core capabilities and fast convergence during compression.

Abstract

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and D2-MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%.
Paper Structure (23 sections, 28 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 23 sections, 28 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Motivation
  3. Mixture-of-Experts Architecture
  4. SVD-based compression with basis sharing for MoE
  5. Motivation
  6. Method
  7. Routing Frequency of Expert Groups
  8. Effective Rank for Expert Group Information Density Evaluation
  9. Adaptive Rank Allocation via Experts Frequency and Information Density
  10. Residual Reconstruction
  11. Experiments
  12. Main Results
  13. Ablation Study
  14. Impact of $\xi$.
  15. Related Work
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Routing frequency of experts for different layers of Qwen3-30B-A3B-2507 from layer 24 to 48.
  • Figure 2: Residual singular value spectrum of gate matrix among experts at layer 0 in Qwen3-30B-A3B-2507. Each column corresponds to one expert, and each row represents the singular-value rank value of $\Delta W$.
  • Figure 3: Language modeling performance of our method on Qwen3-235B-A22B-2507. Perplexity ($\downarrow$) is reported on WikiText-2, PTB, and C4.
  • Figure 4: Training scaled loss comparison between MoBE and RFID-MoE (Ours) for 47-th layer of Qwen3-30B-A3B-2507. (a) Gate projection matrix. (b) Up projection matrix