Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

Tuan Minh Pham, Thinh Cao, Viet Nguyen, Huy Nguyen, Nhat Ho, Alessandro Rinaldo

TL;DR

This work studies sigmoid-gated multinomial MoEs (MLMoE) for classification, addressing three gaps: lack of theoretical guarantees in classification, convergence under over-specification, and temperature effects. It introduces a modified sigmoid gating with an extra positive scale to ensure density convergence and derives density-estimation and Voronoi-loss–based parameter rates, showing sigmoid gating is more sample-efficient than softmax under overspecification. It then analyzes the temperature parameter, showing that a naive inner-product affinity with temperature induces unfavorable interactions and exponential sample complexity, while a Euclidean affinity score removes this interaction and recovers polynomial-rate convergence. Theoretical results are corroborated by numerical experiments demonstrating faster convergence for sigmoid gating without temperature, slower convergence with temperature (inner product), and improved rates with Euclidean affinity, offering practical design guidance for scalable MoE classifiers.

Abstract

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.

Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

TL;DR

This work studies sigmoid-gated multinomial MoEs (MLMoE) for classification, addressing three gaps: lack of theoretical guarantees in classification, convergence under over-specification, and temperature effects. It introduces a modified sigmoid gating with an extra positive scale to ensure density convergence and derives density-estimation and Voronoi-loss–based parameter rates, showing sigmoid gating is more sample-efficient than softmax under overspecification. It then analyzes the temperature parameter, showing that a naive inner-product affinity with temperature induces unfavorable interactions and exponential sample complexity, while a Euclidean affinity score removes this interaction and recovers polynomial-rate convergence. Theoretical results are corroborated by numerical experiments demonstrating faster convergence for sigmoid gating without temperature, slower convergence with temperature (inner product), and improved rates with Euclidean affinity, offering practical design guidance for scalable MoE classifiers.

Abstract

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.
Paper Structure (25 sections, 8 theorems, 131 equations, 1 figure, 4 tables)

This paper contains 25 sections, 8 theorems, 131 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modified Sigmoid Gating Function
  4. Sigmoid Gating Function with Temperature
  5. Inner product affinity score
  6. Euclidean affinity score
  7. Numerical Experiments
  8. Sigmoid gate versus Softmax gate in the MLMoE
  9. Sigmoid gating function with temperature
  10. Discussion
  11. Proofs of Main Results
  12. Proof of Theorem \ref{['theorem:without_temperature']}
  13. Proof of Theorem \ref{['theorem:with_temperature_inner']}
  14. Main Proof
  15. Proof of Lemma A.1.
  16. ...and 10 more sections

Key Result

Proposition 1

(Identifiability) Assume that $G$ and $G'$ are two mixing measures in $\mathcal{G}_k(\Theta)$. If $p_G(Y|X) = p_{G'}(Y|X)$ with probability one (with respect to the joint distribution of $X$ and $Y$), then $G \equiv G'$.

Figures (1)

  • Figure 1: Log-log scaled plots for the empirical convergence rates. Left column: comparison between sigmoid gating without temperature ($G_n$) and softmax gating for the Voronoi metric. Middle and right columns: convergence rates of the MLE $\widetilde{G}_n$ and $\overline{G}_n$ for inner product affinity score and Euclidean affinity score settings, respectively. In these figures, the corresponding empirical discrepancies are illustrated by the blue curves for sigmoid gating, while the red curves represent softmax gating. Orange dash-dotted lines indicate the least-squares fitted linear regression lines. Error bars represent two times the empirical standard deviation.

Theorems & Definitions (9)

  • Proposition 1
  • Proposition 2
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 1
  • Lemma 2
  • proof : Proof of Lemma \ref{['lemma:n-covering']}
  • Lemma 3