DynaMo: Runtime Switchable Quantization for MoE with Cross-Dataset Adaptation
Zihao Zheng, Xiuping Cui, Size Zheng, Maoliang Li, Jiayu Chen, Yun Liang, Xiang Chen
TL;DR
DynaMo tackles the challenge of quantizing Mix-of-Experts (MoE) models across diverse datasets by identifying MoE dynamics at multiple levels and leveraging cross-dataset adaptation. It combines an expert-level mixed-precision baseline with a channel-level dynamic switching mechanism guided by synthesized cross-dataset expert significance, enabling robust performance across unseen data. Empirical results show substantial perplexity reductions and accuracy gains at ~3-bit precision, along with 2.9–3.1× inference speedups and modest overhead. This approach provides a practical, dataset-aware quantization framework that maintains MoE flexibility while delivering efficient, accurate inference in varied data regimes.
Abstract
As the Mix-of-Experts (MoE) architecture increases the number of parameters in large models, there is an even greater need for model quantization. However, existing quantization methods overlook the expert dynamics of MoE across multiple datasets. Moreover, the existing static quantization cannot adapt MoE to various data change scenarios. In this paper, we perform a multi-level analysis to reveal MoE dynamics and define the significance of each channel/each expert. Based on the analysis results, we propose \textit{DynaMo}, an end-to-end MoE quantization framework. DynaMo adopts an expert-level mixed-precision baseline quantization strategy, which ensures the quantized MoEs are compatible with multiple existing datasets. Furthermore, DynaMo incorporates a channel-level dynamic switching mechanism to adapt these quantized MoE models to novel datasets. Experiments show that DynaMo achieves a 2.78~4.54 PPL decrease and a 1.85%~3.77% accuracy improvement in various datasets, with ~3x inference speedup and negligible overhead.