Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Compression for Large-Scale MoE Deployment on Edge

Yixiao Chen, Yanyue Xie, Ruining Yang, Wei Jiang, Wei Wang, Yong He, Yue Chen, Pu Zhao, Yanzhi Wang

TL;DR

The paper tackles the memory bottleneck of ultra-large MoE models for edge deployment by proposing a collaborative compression framework that jointly employs Performance-Aware Expert Pruning, Hardware-Aware Activation Adjustment, and Mixed-Precision Quantization. This pipeline prunes underutilized experts, adapts activation routing, and performs tensor-level sensitivity-guided quantization upgrades within a memory budget, aided by a dynamic budgeting and back-off scheme. On DeepSeek-V3 (671B parameters), the approach reduces storage from 1.3TB to 103GB and delivers higher accuracy than uniform low-bit quantization across multiple benchmarks, including BBH, MMLU, and GSM8K, with feasible edge-device latency. The work enables practical deployment of MoE-scale models on devices with around 128GB of memory, illustrating both theoretical and real-world viability for edge AI.

Abstract

The Mixture of Experts (MoE) architecture is an important method for scaling Large Language Models (LLMs). It increases model capacity while keeping computation cost low. However, the ultra-large MoE models still have hundreds of billions of parameters, requiring massive memory/storage and leading to difficulties for deployment on resource-constrained edge platforms. Pruning or quantization alone can hardly address the issue, because of the super-aggressive compression ratio with significantly degraded accuracy and output quality. To facilitate the deployment of ultra-large MoEs on edge platforms, we propose a collaborative compression framework by combining expert pruning, mixed-precision quantization, and activation optimization. It can effectively reduce the storage footprint of the ultra-large MoE DeepSeek-V3 from 1.3TB to 103GB, while preserving high output quality with better accuracy than traditional uniform low-bit quantization methods. To the best of our knowledge, we are the first to deploy a compressed model from the ultra-large DeepSeek-V3 on the platform with a strict 128GB total memory limit. Our comprehensive experiments on multiple benchmarks under various memory constraints demonstrate the effectiveness of our method with smaller model sizes and higher accuracy than uniform low-bit quantization methods.

Collaborative Compression for Large-Scale MoE Deployment on Edge

TL;DR

The paper tackles the memory bottleneck of ultra-large MoE models for edge deployment by proposing a collaborative compression framework that jointly employs Performance-Aware Expert Pruning, Hardware-Aware Activation Adjustment, and Mixed-Precision Quantization. This pipeline prunes underutilized experts, adapts activation routing, and performs tensor-level sensitivity-guided quantization upgrades within a memory budget, aided by a dynamic budgeting and back-off scheme. On DeepSeek-V3 (671B parameters), the approach reduces storage from 1.3TB to 103GB and delivers higher accuracy than uniform low-bit quantization across multiple benchmarks, including BBH, MMLU, and GSM8K, with feasible edge-device latency. The work enables practical deployment of MoE-scale models on devices with around 128GB of memory, illustrating both theoretical and real-world viability for edge AI.

Abstract

The Mixture of Experts (MoE) architecture is an important method for scaling Large Language Models (LLMs). It increases model capacity while keeping computation cost low. However, the ultra-large MoE models still have hundreds of billions of parameters, requiring massive memory/storage and leading to difficulties for deployment on resource-constrained edge platforms. Pruning or quantization alone can hardly address the issue, because of the super-aggressive compression ratio with significantly degraded accuracy and output quality. To facilitate the deployment of ultra-large MoEs on edge platforms, we propose a collaborative compression framework by combining expert pruning, mixed-precision quantization, and activation optimization. It can effectively reduce the storage footprint of the ultra-large MoE DeepSeek-V3 from 1.3TB to 103GB, while preserving high output quality with better accuracy than traditional uniform low-bit quantization methods. To the best of our knowledge, we are the first to deploy a compressed model from the ultra-large DeepSeek-V3 on the platform with a strict 128GB total memory limit. Our comprehensive experiments on multiple benchmarks under various memory constraints demonstrate the effectiveness of our method with smaller model sizes and higher accuracy than uniform low-bit quantization methods.

Paper Structure

This paper contains 10 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Performance-Aware Expert Pruning
  5. Hardware-Aware Activation Adjustment
  6. Mixed-Precision Quantization
  7. Experiments
  8. Experiments Settings
  9. Results
  10. Conclusion

Figures (3)

  • Figure 1: Illustration of our proposed MoE compression framework. The middle part shows a block-level schematic of DeepSeek MoE. The left part highlights our Performance-Aware Expert Reduction and Pruning-Aware Activation Adjustment for FFN layers, while the right part illustrates the mixed-precision quantization process applied to the remaining model.
  • Figure 2: Visualization of Expert Activation Count and Cumulative Activation Score in the 1st and last MoE Layer of DeepSeek-V3. Results on the calibration dataset reveal a pronounced imbalance in both activation frequency and contribution across experts, thereby justifying the feasibility of our expert pruning strategy.
  • Figure 3: DeepSeek-V3 benchmark comparision of llamacpp unified IQ1_M quant (140G) and ours smallest compression 103GB model.