Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expert Merging in Sparse Mixture of Experts with Nash Bargaining

Dung V. Nguyen, Anh T. Nguyen, Minh H. Nguyen, Luc Q. Nguyen, Shiqi Jiang, Ethan Fetaya, Linh Duy Tran, Gal Chechik, Tan M. Nguyen

TL;DR

This work addresses the challenge of merging experts in Sparse MoE by introducing NAMEx, which casts merging as a Nash Bargaining problem to achieve fair, coordinated contributions from all experts. It further augments NAMEx with complex momentum (NAMEx-Momentum) to accelerate convergence and provide theoretical guarantees. Empirical results across language modeling, text classification, image classification, and zero-shot robustness show NAMEx and its momentum-enhanced variants consistently outperform existing merging methods, including in large-scale deployments such as Qwen1.5-MoE (14B) and DeepSeek-MoE (16B). The approach enables scalable, robust, and interpretable expert collaboration in SMoE, with practical implications for deploying large, modular transformers while maintaining efficiency.

Abstract

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.

Expert Merging in Sparse Mixture of Experts with Nash Bargaining

TL;DR

This work addresses the challenge of merging experts in Sparse MoE by introducing NAMEx, which casts merging as a Nash Bargaining problem to achieve fair, coordinated contributions from all experts. It further augments NAMEx with complex momentum (NAMEx-Momentum) to accelerate convergence and provide theoretical guarantees. Empirical results across language modeling, text classification, image classification, and zero-shot robustness show NAMEx and its momentum-enhanced variants consistently outperform existing merging methods, including in large-scale deployments such as Qwen1.5-MoE (14B) and DeepSeek-MoE (16B). The approach enables scalable, robust, and interpretable expert collaboration in SMoE, with practical implications for deploying large, modular transformers while maintaining efficiency.

Abstract

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings.
Paper Structure (39 sections, 3 theorems, 40 equations, 11 figures, 13 tables, 2 algorithms)

This paper contains 39 sections, 3 theorems, 40 equations, 11 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Sparse Mixture of Experts
  4. Curvature-aware Merging of Experts
  5. Nash Bargaining in Multi-Task Learning
  6. Nash Merging of Experts
  7. Merging Experts as a Bargaining Game
  8. NAMEx as the Nash Solution of Expert Merging
  9. Integrating Momentum Into Expert Merging
  10. Experimental Results
  11. Language Modeling
  12. Text Classification
  13. Image Classification
  14. Zero-shot and Finetuning on DeepSeek-MoE (16B) and Qwen1.5-MoE (14B)
  15. Empirical Analysis
  16. ...and 24 more sections

Key Result

Lemma 3.1

Let $\mathbf{G}$ denote the $d \times N$ matrix whose columns are the domain-vectors $\tau_i$. The solution to is (up to scaling) $\Delta\boldsymbol{\mathcal{E}}^* = \sum_{i=1}^N \alpha_i \tau_i$, where $\alpha \in \mathbb{R}^N_+$ satisfies $\mathbf{G}^\top \mathbf{G} \alpha = 1 / \alpha$, with $1 / \alpha$ being the element-wise reciprocal.

Figures (11)

  • Figure 1: Cosine similarity of expert outputs in Swin-MoE liu2021Swin (top), Switch-Transformer fedus2022switch (middle), and Qwen-MoE qwen2 (bottom). Swin-MoE shows stable mid-layer features, Switch-Transformer exhibits dynamic routing at Layer 8, and Qwen-MoE yields robust final representations at Layer 9--highlighting diverse expert interaction patterns.
  • Figure 2: Architecture overview of (a) CAMEx iclr2024camex, (b) Expert-Propagation CAMEx iclr2024camex, and (c) our proposed merging method, NAMEx.
  • Figure 3: Top-1 Accuracy Evaluation of Swin Transformer Variants. Complex momentum enhances convergence speed and improves the performance of both NAMEx and EP-CAMEx.
  • Figure 4: Graph of system of inequality \ref{['eq:ineq']} when $\boldsymbol{\alpha} = 0.5$ and $\gamma = 2$.
  • Figure 5: L2 distance between expert updates across training steps (T5-Base, 12 MoE layers). Lower values indicate better stability. The figure shows that NAMEx converges faster and more stably than EP-CAMEx.
  • ...and 6 more figures

Theorems & Definitions (8)

  • Lemma 3.1: Nash Solution of Expert Merging
  • Definition 3.2: NAMEx: Nash Merging of Experts
  • Definition 3.3: NAMEx-Momentum: Nash Merging with Complex Momentum
  • Proposition 3.3: Convergence rate of NAMEx-Momentum
  • proof
  • Theorem B.3: Consequence of Prop. 4.4.1 bertsekas2008nonlinear
  • proof
  • proof