Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts

TL;DR

The paper advances the theoretical understanding of Gaussian-gated Mixture of Experts by establishing a parametric-density convergence rate for the MLE and by deriving detailed parameter-estimation rates under two complementary gating-parametric regimes (Type I and Type II). It introduces Voronoi-based losses that link density accuracy to component-wise parameter rates and shows that exact-fitted components achieve the standard $O(n^{-1/2})$ rate, while over-fitted components exhibit slower rates governed by solvability of associated polynomial systems. In Type I, interior PDE interactions drive the slower rates for some parameters, whereas Type II introduces an exterior interaction that further alters the rates, particularly for $a_j^0$. A density-rate bound $V(p_{\hat{G}_n},p_{G_0})=O(\sqrt{\log n / n})$ holds in both settings, and simulation experiments corroborate the theoretical rates, illustrating practical insights for over-fitting and component-count selection in GMoE models.

Abstract

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications of machine learning and statistics. Despite its popularity in practice, a satisfactory level of theoretical understanding of the MoE model is far from complete. To shed new light on this problem, we provide a convergence analysis for maximum likelihood estimation (MLE) in the Gaussian-gated MoE model. The main challenge of that analysis comes from the inclusion of covariates in the Gaussian gating functions and expert networks, which leads to their intrinsic interaction via some partial differential equations with respect to their parameters. We tackle these issues by designing novel Voronoi loss functions among parameters to accurately capture the heterogeneity of parameter estimation rates. Our findings reveal that the MLE has distinct behaviors under two complement settings of location parameters of the Gaussian gating functions, namely when all these parameters are non-zero versus when at least one among them vanishes. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to empirically verify our theoretical results.

Figures (3)

• Figure 1: Log-log scaled plots of the empirical mean of discrepancies $\overline{D}(\widehat{G}_n,G_0)$ and $\widetilde{D}(\widehat{G}_n,G_0)$ and $G_0$ (orange lines with error bars) and least-squares fitted linear regression (black dash-dotted lines) when $d=1$ and $k_0 = 3$.
• Figure 2: Log-log scaled plots of the empirical mean of discrepancies $\overline{D}(\widehat{G}_n,G_0)$ and $\widetilde{D}(\widehat{G}_n,G_0)$ (orange lines with error bars) and least-squares fitted linear regression (black dash-dotted lines) when $d=2$ and $k_0 = 3$.
• Figure 3: Illustration of Voronoi cells defined in equation \ref{['eq_new_Voronoi_cell']} when $k_0 = 6$ and $k = 10$. In this figure, red squares represent for true components (i.e. components of $G_0$), while blue circles indicate fitted components (i.e. components of $G$). By definition, each Voronoi cell is generated by one true component, and its cardinality is exactly the number of corresponding fitted components. For example, the square in cell 4 is approximated by two rounds, which means that the cardinality of cell 4 is two.

Theorems & Definitions (8)

• Proposition 1: Identifiability of the GMoE model
• Proposition 2: Joint Density Estimation Rate
• Lemma 1: Proposition 2.1, ho_convergence_2016
• Theorem 1
• Lemma 2
• Theorem 2
• Lemma 3: Theorem 7.4, Vandegeer-2000
• Lemma 4