MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

TL;DR

MoEQuant tackles the quantization challenges of Mixture-of-Experts LLMs by identifying inter- and intra-expert imbalances during calibration. It introduces Expert-Balanced Self-Sampling (EBSS) to generate calibration data distributed evenly across experts and Affinity-Guided Quantization (AGQ) to weight quantization errors by token-expert affinities, making PTQ more accurate for MoE. Across three MoE LLMs and multiple tasks, MoEQuant delivers substantial gains at 4-bit quantization (e.g., >10 points on HumanEval) and achieves memory savings and speedups, enabling deployment on consumer GPUs. The approach is plug-and-play with existing PTQ methods and improves generalization, especially for instruction-tuned MoE variants.

Abstract

Mixture-of-Experts (MoE) large language models (LLMs), which leverage dynamic routing and sparse activation to enhance efficiency and scalability, have achieved higher performance while reducing computational costs. However, these models face significant memory overheads, limiting their practical deployment and broader adoption. Post-training quantization (PTQ), a widely used method for compressing LLMs, encounters severe accuracy degradation and diminished generalization performance when applied to MoE models. This paper investigates the impact of MoE's sparse and dynamic characteristics on quantization and identifies two primary challenges: (1) Inter-expert imbalance, referring to the uneven distribution of samples across experts, which leads to insufficient and biased calibration for less frequently utilized experts; (2) Intra-expert imbalance, arising from MoE's unique aggregation mechanism, which leads to varying degrees of correlation between different samples and their assigned experts. To address these challenges, we propose MoEQuant, a novel quantization framework tailored for MoE LLMs. MoE-Quant includes two novel techniques: 1) Expert-Balanced Self-Sampling (EBSS) is an efficient sampling method that efficiently constructs a calibration set with balanced expert distributions by leveraging the cumulative probabilities of tokens and expert balance metrics as guiding factors. 2) Affinity-Guided Quantization (AGQ), which incorporates affinities between experts and samples into the quantization process, thereby accurately assessing the impact of individual samples on different experts within the MoE layer. Experiments demonstrate that MoEQuant achieves substantial performance gains (more than 10 points accuracy gain in the HumanEval for DeepSeekMoE-16B under 4-bit quantization) and boosts efficiency.

Figures (5)

• Figure 1: The relative accuracy gains of GPTQ across various models for two generative tasks, HumanEval and GSM8k, before and after applying MoEQuant are presented. The suffix "I" denotes the instruction fine-tuned version.
• Figure 2: Sample distribution on the first MoE layer of Qwen-MoE-A2.7B-14B for different calibration sets. For C4 and WikiText2, 128 × 512 tokens were sampled, for our EBSS, samples were generated through the model's self-sampling method.
• Figure 3: The MoE structure in LLMs. The router selects all non-shared experts and $k$ shared experts with highest confidence. The predictions from all experts are then aggregated and weighted.
• Figure 4: Perplexity performance on DeepSeek-MoE-16B of different datasets.
• Figure 5: Illustrative diagram of EBSS. The expert distribution and cumulative probabilities jointly guide the path searching.