Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures

TL;DR

The paper tackles the challenging problem of selecting the number of experts in Gaussian-gated Gaussian MoEs (GGMoEs) when covariates influence both gating and experts. It extends the concept of dendrograms of mixing measures and introduces the Dendrogram Selection Criterion (DSC), a tuning-parameter-robust method that can consistently recover the true component count and achieve fast parameter convergence for merged experts in overfitted models. A Voronoi-based loss and inverse bounds underpin the theoretical guarantees, and the DSC approach avoids exhaustive model fitting across different K by emitting a hierarchical merging procedure from an overfitted MLE. Empirical studies on synthetic data show DSC outperforms AIC, BIC, and ICL in identifying the correct number of components and in accurate parameter and regression function recovery, offering a scalable, interpretable alternative for heterogeneous data analysis in high-dimensional or deep learning applications.

Abstract

Mixture of Experts (MoE) models constitute a widely utilized class of ensemble learning approaches in statistics and machine learning, known for their flexibility and computational efficiency. They have become integral components in numerous state-of-the-art deep neural network architectures, particularly for analyzing heterogeneous data across diverse domains. Despite their practical success, the theoretical understanding of model selection, especially concerning the optimal number of mixture components or experts, remains limited and poses significant challenges. These challenges primarily stem from the inclusion of covariates in both the Gaussian gating functions and expert networks, which introduces intrinsic interactions governed by partial differential equations with respect to their parameters. In this paper, we revisit the concept of dendrograms of mixing measures and introduce a novel extension to Gaussian-gated Gaussian MoE models that enables consistent estimation of the true number of mixture components and achieves the pointwise optimal convergence rate for parameter estimation in overfitted scenarios. Notably, this approach circumvents the need to train and compare a range of models with varying numbers of components, thereby alleviating the computational burden, particularly in high-dimensional or deep neural network settings. Experimental results on synthetic data demonstrate the effectiveness of the proposed method in accurately recovering the number of experts. It outperforms common criteria such as the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood, while achieving optimal convergence rates for parameter estimation and accurately approximating the regression function.

Figures (4)

• Figure 1: Convergence rates of overfitted, exact-fitted, and merged mixing measures for GGMoE.
• Figure 2: Dendrogram mixing measures with $K = 10$ and $K_0 = 3$, arranged in a clockwise layout to enhance visual comparison between the merged estimates and the true mixing measure.
• Figure 3: Model selection with $K =20, K_0 = 3$, $N_{\mathop{\mathrm{num}}\limits} = 64$, $N_{\min} = 200$, $N_{\max} = 2*10^4$, $N_{\mathop{\mathrm{rep}}\limits} = 80$.
• Figure 4: Typical simulation of data set from GGMoE model.

Theorems & Definitions (9)

• Lemma 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• proof : Proof of \ref{['lemma_r_bar_GGMoE']}
• Lemma 11: Condition K for GGMoE model
• proof : Proof of \ref{['lem_condition_K']}