Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge

Shengling Qin, Hai Wu, Hongyang Du, Kaibin Huang

TL;DR

This work tackles energy-efficient expert coordination for distributed Mixture-of-Experts at the wireless edge by formulating two intertwined optimization problems: expert selection (P1) and joint expert with subcarrier allocation (P2). It introduces Dynamic Expert Selection (DES), a tree-search framework with a linear-relaxation bound, to balance task-relevance and channel conditions, and Joint Expert and Subcarrier Allocation (JESA), a block coordinate descent approach that leverages a unique problem structure to achieve asymptotic optimality as subcarriers increase. Empirical results on multi-domain tasks with Llama-3-based experts show DES delivers high accuracy with significantly reduced energy, while JESA achieves substantial energy savings (often around 50% or more) with modest accuracy trade-offs, outperforming Top-k and homogeneous schemes. Collectively, the framework advances integrated AI-and-communications at the edge, offering tunable tradeoffs via layer-wise importance and enabling scalable, energy-conscious edge inference for future 6G networks.

Abstract

The emergence of distributed Mixture-of-Experts (DMoE) systems, which deploy expert models at edge nodes, offers a pathway to achieving connected intelligence in sixth-generation (6G) mobile networks and edge artificial intelligence (AI). However, current DMoE systems lack an effective expert selection algorithm to address the simultaneous task-expert relevance and channel diversity inherent in these systems. Traditional AI or communication systems focus on either performance or channel conditions, and direct application of these methods leads to high communication overhead or low performance. To address this, we propose the DMoE protocol to schedule the expert inference and inter-expert transmission. This protocol identifies expert selection and subcarrier allocation as key optimization problems. We formulate an expert selection problem by incorporating both AI performance and channel conditions, and further extend it to a Joint Expert and Subcarrier Allocation (JESA) problem for comprehensive AI and channel management within the DMoE framework. For the NP-hard expert selection problem, we introduce the Dynamic Expert Selection (DES) algorithm, which leverages a linear relaxation as a bounding criterion to significantly reduce search complexity. For the JESA problem, we discover a unique structural property that ensures asymptotic optimality in most scenarios. We propose an iterative algorithm that addresses subcarrier allocation as a subproblem and integrates it with the DES algorithm. The proposed framework effectively manages the tradeoff between task relevance and channel conditions through a tunable importance factor, enabling flexible adaptation to diverse scenarios. Numerical experiments validate the dual benefits of the proposed expert selection algorithm: high performance and significantly reduced cost.

Optimal Expert Selection for Distributed Mixture-of-Experts at the Wireless Edge

TL;DR

This work tackles energy-efficient expert coordination for distributed Mixture-of-Experts at the wireless edge by formulating two intertwined optimization problems: expert selection (P1) and joint expert with subcarrier allocation (P2). It introduces Dynamic Expert Selection (DES), a tree-search framework with a linear-relaxation bound, to balance task-relevance and channel conditions, and Joint Expert and Subcarrier Allocation (JESA), a block coordinate descent approach that leverages a unique problem structure to achieve asymptotic optimality as subcarriers increase. Empirical results on multi-domain tasks with Llama-3-based experts show DES delivers high accuracy with significantly reduced energy, while JESA achieves substantial energy savings (often around 50% or more) with modest accuracy trade-offs, outperforming Top-k and homogeneous schemes. Collectively, the framework advances integrated AI-and-communications at the edge, offering tunable tradeoffs via layer-wise importance and enabling scalable, energy-conscious edge inference for future 6G networks.

Abstract

The emergence of distributed Mixture-of-Experts (DMoE) systems, which deploy expert models at edge nodes, offers a pathway to achieving connected intelligence in sixth-generation (6G) mobile networks and edge artificial intelligence (AI). However, current DMoE systems lack an effective expert selection algorithm to address the simultaneous task-expert relevance and channel diversity inherent in these systems. Traditional AI or communication systems focus on either performance or channel conditions, and direct application of these methods leads to high communication overhead or low performance. To address this, we propose the DMoE protocol to schedule the expert inference and inter-expert transmission. This protocol identifies expert selection and subcarrier allocation as key optimization problems. We formulate an expert selection problem by incorporating both AI performance and channel conditions, and further extend it to a Joint Expert and Subcarrier Allocation (JESA) problem for comprehensive AI and channel management within the DMoE framework. For the NP-hard expert selection problem, we introduce the Dynamic Expert Selection (DES) algorithm, which leverages a linear relaxation as a bounding criterion to significantly reduce search complexity. For the JESA problem, we discover a unique structural property that ensures asymptotic optimality in most scenarios. We propose an iterative algorithm that addresses subcarrier allocation as a subproblem and integrates it with the DES algorithm. The proposed framework effectively manages the tradeoff between task relevance and channel conditions through a tunable importance factor, enabling flexible adaptation to diverse scenarios. Numerical experiments validate the dual benefits of the proposed expert selection algorithm: high performance and significantly reduced cost.

Paper Structure

This paper contains 29 sections, 4 theorems, 24 equations, 10 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. System Model
  3. Communication Model
  4. Energy Consumption Models
  5. Overview of Distributed Mixture-of-Experts
  6. System Initialization
  7. Expertise Diversity
  8. DMoE Protocol
  9. Problem Formulation
  10. Expert Selection Problem Formulation
  11. Joint Expert and Subcarrier Allocation Problem Formulation
  12. Optimal Expert Selection
  13. Problem Reformulation
  14. Search Tree Construction and Breadth-First Search
  15. Bounding Criterion
  16. ...and 14 more sections

Key Result

Proposition 1

Problem P1(a) is NP-hard.

Figures (10)

  • Figure 1: (a) The DMoE system; (b) The DMoE protocol.
  • Figure 2: The architecture of a Mixture-of-Expert model.
  • Figure 3: The normalized performance of the MoE model and individual experts across multi-domain tasks. The expertise diversity is observed.
  • Figure 4: The expert selection process in DMoE.
  • Figure 5: The final accuracy versus the starting layer index with lowered QoS requirement.
  • ...and 5 more figures

Theorems & Definitions (10)

  • Remark 1: Distribution Methods
  • Remark 2
  • Proposition 1
  • proof
  • Theorem 1
  • proof
  • Corollary 1: Asymptotically Optimal
  • Remark 3
  • Proposition 2
  • proof