Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?

TL;DR

This work analyzes how softmax temperature in dense-to-sparse Gaussian MoEs influences maximum-likelihood estimation, revealing potential subpolynomial convergence caused by PDE-driven interactions between temperature and gating/expert parameters. It introduces Activation Dense-to-sparse gating, routing linear-transformed inputs through an activation before softmax, and proves that first- and second-order independence properties of the activation can restore polynomial rates for key parameters under exact- and over-specified settings. The density estimation rate consistently achieves a parametric $\widetilde{\mathcal{O}}(n^{-1/2})$, while original gating interactions can yield as slow as $\mathcal{O}(1/\log(n))$ for some parameters; the activated gate mitigates these issues, yielding $\widetilde{\mathcal{O}}(n^{-1/2})$ for gating and often $\widetilde{\mathcal{O}}(n^{-1/4})$ or $\widetilde{\mathcal{O}}(n^{-1/2})$ for others depending on the setting. Numerical experiments corroborate the theory, showing substantial rate improvements with activation gates and highlighting implications for expert selection, misspecification, and MoE design. The results offer a principled path to more sample-efficient MoE training by choosing gating mechanisms that suppress detrimental parameter interactions without sacrificing density-fit quality.

Abstract

Dense-to-sparse gating mixture of experts (MoE) has recently become an effective alternative to a well-known sparse MoE. Rather than fixing the number of activated experts as in the latter model, which could limit the investigation of potential experts, the former model utilizes the temperature to control the softmax weight distribution and the sparsity of the MoE during training in order to stabilize the expert specialization. Nevertheless, while there are previous attempts to theoretically comprehend the sparse MoE, a comprehensive analysis of the dense-to-sparse gating MoE has remained elusive. Therefore, we aim to explore the impacts of the dense-to-sparse gate on the maximum likelihood estimation under the Gaussian MoE in this paper. We demonstrate that due to interactions between the temperature and other model parameters via some partial differential equations, the convergence rates of parameter estimations are slower than any polynomial rates, and could be as slow as $\mathcal{O}(1/\log(n))$, where $n$ denotes the sample size. To address this issue, we propose using a novel activation dense-to-sparse gate, which routes the output of a linear layer to an activation function before delivering them to the softmax function. By imposing linearly independence conditions on the activation function and its derivatives, we show that the parameter estimation rates are significantly improved to polynomial rates. Finally, we conduct a simulation study to empirically validate our theoretical results.

Figures (2)

• Figure 1: Log-log scaled plots for the empirical convergence rates of the MLE $\widehat{G}_n$ for exact and over-specified settings. In these figures, the corresponding empirical discrepancies are illustrated by the blue curves, while the orange dash-dotted lines represent for the least-squares fitted linear regression lines. The error bars represent two times the empirical standard deviation under the exact-specified setting.
• Figure 2: A visual illustration depicting the correlation between variables $X$ and $Y$, along with their individual marginal distributions.

Theorems & Definitions (18)

• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Theorem 2.4
• Theorem 2.5
• Theorem 3.1
• Definition 3.2: First-order Independence
• Theorem 3.3
• Definition 3.4: Second-order Independence
• Theorem 3.5