Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation

Zhenyu Liu, Yunzhen Liu, Zehao Fan, Garrett Gagnon, Yayue Hou, Nan Wu, Yangwook Kang, Liu Liu

TL;DR

MoE inference faces memory bandwidth bottlenecks when offloading large expert sets. The authors introduce a router-guided, low-rank compensation framework that offline-computes per-expert low-rank corrections and online-selects Top-n experts per token for compensation, preserving low-bit transfers for the rest. This joint offline-online design achieves strong bandwidth-accuracy trade-offs and notable throughput gains on both GPU-only and GPU-NDP deployments, with ablations validating the effectiveness of kurtosis-guided rank allocation and selective restoration. The approach offers a practical path to deploying larger MoE models in bandwidth-constrained environments with minimal accuracy loss.

Abstract

Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput.

Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation

TL;DR

MoE inference faces memory bandwidth bottlenecks when offloading large expert sets. The authors introduce a router-guided, low-rank compensation framework that offline-computes per-expert low-rank corrections and online-selects Top-n experts per token for compensation, preserving low-bit transfers for the rest. This joint offline-online design achieves strong bandwidth-accuracy trade-offs and notable throughput gains on both GPU-only and GPU-NDP deployments, with ablations validating the effectiveness of kurtosis-guided rank allocation and selective restoration. The approach offers a practical path to deploying larger MoE models in bandwidth-constrained environments with minimal accuracy loss.

Abstract

Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput.

Paper Structure

This paper contains 15 sections, 6 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Mixture-of-Experts Architecture
  4. Dynamics of MoE Inference
  5. Low-rank Compensation for Experts
  6. Method
  7. Kurtosis-Guided Rank Allocation
  8. Router-Guided Error Compenstation
  9. Evaluation
  10. Experimental Setup
  11. Accuracy Evaluation
  12. Implications on System Performance
  13. Ablation Study
  14. Related Work
  15. Conclusion

Figures (8)

  • Figure 1: MoE inference time breakdown and roofline.
  • Figure 2: Decoding expert router patterns.
  • Figure 3: Router score distribution on C4 calibration dataset.
  • Figure 4: (a) Residual errors introduced by low-bit quantization, which can be effectively restored using a low-rank compensator. (b) Relationship between kurtosis and quantization error, showing that kurtosis is positively related to quantization error, so that higher-kurtosis experts require higher-rank compensation.
  • Figure 5: Overview of our method.
  • ...and 3 more figures