PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

TL;DR

PuzzleMoE addresses the high memory overhead of large Mixture-of-Experts models by presenting a training-free compression method that performs sparse, fine-grained expert merging guided by a pair of masks to retain both shared and expert-specific weights. It introduces a bit-packed inference scheme that embeds masks and signs into weight representations, enabling metadata-free, GPU-accelerated inference with a custom on-the-fly decoding CUDA kernel. Empirically, PuzzleMoE achieves up to 50% compression with minimal accuracy loss across multiple MoE models and tasks, outperforming prior MoE compression methods and delivering up to $1.28\times$ inference speedup. The approach combines hardware-aware weight encoding with a pairwise dual-mask merging strategy to provide a practical, scalable solution for deploying large MoE models on constrained hardware.

Abstract

Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies, yet they often suffer from performance drop at high compression ratios. In this paper, we introduce PuzzleMoE, a training-free MoE compression method that achieves both high accuracy and efficient inference through two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It uses a dual-mask to capture both shared and expert-specific parameters. Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE can compress MoE models by up to 50% while maintaining accuracy across various tasks. Specifically, it outperforms prior MoE compression methods by up to 16.7% on MMLU at 50% compression ratio, and achieves up to 1.28\times inference speedup.

Figures (8)

• Figure 1: (a): Accuracy of different MoE models on MMLU benchmark under 50%compression ratio with various expert compression methods, among which PuzzleMoE achieves the best accuracy. (b): Comparison of different expert compression methods, among which PuzzleMoE effectively and efficiently retains MoE models performance after compression with task-agnostic design.
• Figure 2: Procedure of sparse expert merging algorithm. We merge two experts by computing saliency-based masks to keep only important weights for each expert, a cross-expert similarity mask to keep aligned weights, and extracting weights' signs. Then, the merged weights $\mathbf{W}_{\mathrm{merged}}$ are formed via Eq. (7). Finally, we store $\mathbf{M}_i$, $\mathbf{M}_j$, $\mathbf{S}_i$, $\mathbf{S}_j$, and $\mathbf{W}_{\mathrm{merged}}$ for inference usage. Masks and signs are bit-packed for compact storage as demonstrated in Section \ref{['subsec:merging-algo']}.
• Figure 3: Illustration of the mask packing procedure. (a): the distribution of Bfloat16 weight exponents in Mixtral-8x7B. After shifting, the exponents can be encoded in 5 bits. (b): bit-level organization of masks and signs within the packed Bfloat16 format.
• Figure 4: System performance for different MoE models and tasks. (a): Compression time comparison across Mixtral and Deepseek-MoE. (b): Memory usage during inference for Mixtral-8x7B and Qwen3-MoE. (c): Inference speedup comparison for Mixtral-8x7B and Qwen3-MoE.
• Figure 5: (a): Wikitext2 perplexity of Mixtral and Deepseek-MoE under different similarity thresholds. (b): Accuracy performance of combining PuzzleMoE with quantization.