Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng

Abstract

Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.

DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge

Abstract

Despite the computational efficiency of MoE models, the excessive memory footprint and I/O overhead inherent in multi-expert architectures pose formidable challenges for real-time inference on resource-constrained edge platforms. While existing static methods struggle with a rigid latency-accuracy trade-off, we observe that expert importance is highly skewed and depth-dependent. Motivated by these insights, we propose DyMoE, a dynamic mixed-precision quantization framework designed for high-performance edge inference. Leveraging insights into expert importance skewness and depth-dependent sensitivity, DyMoE introduces: (1) importance-aware prioritization to dynamically quantize experts at runtime; (2) depth-adaptive scheduling to preserve semantic integrity in critical layers; and (3) look-ahead prefetching to overlap I/O stalls. Experimental results on commercial edge hardware show that DyMoE reduces Time-to-First-Token (TTFT) by 3.44x-22.7x and up to a 14.58x speedup in Time-Per-Output-Token (TPOT) compared to state-of-the-art offloading baselines, enabling real-time, accuracy-preserving MoE inference on resource-constrained edge devices.
Paper Structure (24 sections, 8 equations, 11 figures, 3 tables)

This paper contains 24 sections, 8 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Mixture-of-Experts Architecture
  4. MoE Compression and Offloading
  5. Observations
  6. Dynamic Skewness in Expert Importance
  7. Depth-Aware Sensitivity to Quantization
  8. Inter-layer Activation Similarity and Look-ahead Opportunities
  9. DyMoE Design
  10. System Overview
  11. Phase-Adaptive Expert Importance Estimator
  12. Prefill: Token-Guided Importance
  13. Decode: Gate-Guided Importance
  14. Depth-Aware Precision Scheduling
  15. Dynamic Expert Orchestration Engine
  16. ...and 9 more sections

Figures (11)

  • Figure 1: Pipeline Comparison: DyMoE vs. Two Conventional MoE Baselines.
  • Figure 2: Overview of MoE Structure and its Memory Demands.
  • Figure 3: Performance evaluation of various expert pruning strategies on the C-Eval benchmark across different retention ratios. Random: experts are retained randomly; Token-based: experts are prioritized based on the volume of assigned critical tokens; Equal: applies a uniform pruning ratio across all layers; Depth-based: adjusts the retention ratio dynamically according to layer depth.
  • Figure 4: Comparative Visualization of Expert Routing Distributions: heavy-hitter and general Tokens across Different Inputs.
  • Figure 5: Layer-wise sensitivity of Mixtral-8x7B-Instruct under Int2 quantization, measured on C-Eval and CMMLU benchmarks.
  • ...and 6 more figures