Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SiftMoE: Similarity-Aware Energy-Efficient Expert Selection for Wireless Distributed MoE Inference

Qian Chen, Xianhao Chen, Kaibin Huang

Abstract

Mixture-of-Experts (MoE) architectures leverage sparse activation to enhance the scalability of large language models (LLMs), making them suitable for deployment in resource-constrained edge networks. However, the sheer number of experts often exceeds the memory capacity of individual edge nodes, necessitating wireless distributed MoE (WIDE) inference where experts are spread across multiple edge nodes. In this context, expert selection directly affects communication costs. Motivated by the similarity of experts, we propose SiftMoE, which judiciously selects or skips experts to strike a tradeoff between communication costs and inference accuracy. Specifically, we first establish theoretical bounds on the accuracy degradation resulting from expert replacement or skipping. Based on the bounds, we formulate an energy minimization problem for expert selection in WIDE inference subject to latency and accuracy constraints. In particular, for slow-fading channels, we derive optimal expert selection policies for both single-token decoding and multi-token prefilling. For fast-fading channels, we further extend our scheme to cope with rapidly varying channel conditions. Simulation results demonstrate that SiftMoE significantly reduces energy consumption while maintaining inference accuracy compared with conventional Top-K routing in WIDE systems.

SiftMoE: Similarity-Aware Energy-Efficient Expert Selection for Wireless Distributed MoE Inference

Abstract

Mixture-of-Experts (MoE) architectures leverage sparse activation to enhance the scalability of large language models (LLMs), making them suitable for deployment in resource-constrained edge networks. However, the sheer number of experts often exceeds the memory capacity of individual edge nodes, necessitating wireless distributed MoE (WIDE) inference where experts are spread across multiple edge nodes. In this context, expert selection directly affects communication costs. Motivated by the similarity of experts, we propose SiftMoE, which judiciously selects or skips experts to strike a tradeoff between communication costs and inference accuracy. Specifically, we first establish theoretical bounds on the accuracy degradation resulting from expert replacement or skipping. Based on the bounds, we formulate an energy minimization problem for expert selection in WIDE inference subject to latency and accuracy constraints. In particular, for slow-fading channels, we derive optimal expert selection policies for both single-token decoding and multi-token prefilling. For fast-fading channels, we further extend our scheme to cope with rapidly varying channel conditions. Simulation results demonstrate that SiftMoE significantly reduces energy consumption while maintaining inference accuracy compared with conventional Top-K routing in WIDE systems.
Paper Structure (28 sections, 6 theorems, 39 equations, 8 figures, 2 algorithms)

This paper contains 28 sections, 6 theorems, 39 equations, 8 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. System Model
  3. Local Computing Model
  4. Computation Offloading Model
  5. Error Analysis of Expert Selection
  6. Preliminaries and Assumptions
  7. Effect of Expert Selection on Output Results
  8. Expert Selection Under Slow Fading
  9. Problem Formulation
  10. Decoding Phase with a Single Token
  11. Prefilling Phase with Multiple Tokens
  12. When $K=1$
  13. When $K \geq 1$
  14. Design insights
  15. Joint Expert Selection and adaptive transmission under fast fading
  16. ...and 13 more sections

Key Result

Proposition 1

With expert selection, the expected deviation at MoE layer $\ell$ satisfies where $F_{i,\varphi(i)}^{(\ell)}\left ( z \right )\triangleq \left \| {\rm{FFN}}_{i}^{(\ell)}\left ( z \right ) \right \|_2 \cdot \sqrt{1+\rho_{i,\varphi(i)}^{(\ell)}\left ( z \right )^2-2\rho_{i,\varphi(i)}^{(\ell)}\left ( z \right ) \cos \theta_{i,\varphi(i)}^{(\ell)}\left ( z \right )

Figures (8)

  • Figure 1: An illustration of the WIDE inference system, where a user sends its tokens to the helpers with the required experts for processing. (a) The proposed SiftMoE framework, where "Exp" denotes expert. Here, expert 2 has a similar function but better channel condition compared with expert 3. (b) Operations of inference within an MoE layer.
  • Figure 2: Timeline of SiftMoE inference.
  • Figure 3: Comparison under slow-fading channels for different maximum tolerable errors per layer on switch-base-8 over XSum, where $B_n = 1$ MHz and $T = 0.7$ s.
  • Figure 4: Comparison under slow-fading channels for different maximum tolerable errors per layer on Mixtral-8x7B over CommonsenseQA, where $B_n = 2$ MHz and $T = 0.074$ s.
  • Figure 5: Effect of system constraints on average energy consumption for switch-base-8 over XSum under slow fading. The default values of $B_n$ and $T$ are set to 1 MHz and 0.7 s.
  • ...and 3 more figures

Theorems & Definitions (16)

  • Proposition 1: Expected layer-wise deviation
  • proof
  • Remark 1: Skipping versus replacement
  • Theorem 1: Final output deviation bound
  • proof
  • Corollary 1: Expected per-layer deviation budget
  • Theorem 2
  • proof
  • Remark 2
  • Proposition 2
  • ...and 6 more