Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts

TL;DR

This work addresses learning in large-scale, streaming settings for mixture-of-experts (MoE) models by revisiting an incremental stochastic MM framework that generalizes incremental EM. It develops a robust surrogate-based online MM approach with explicit convergence guarantees, applying it to softmax-gated MoE (SGMoE) models, including polynomial-parameterizations for gating and means. Theoretical results establish almost-sure convergence to stationary points of the expected objective, while empirical studies on synthetic and real-world data show the proposed method outperforms standard stochastic optimizers such as SGD, RMSProp, Adam, and Sophia. The approach enables practical incremental learning for complex MoE architectures and paves the way for extensions to non-Gaussian experts, mini-batch updates, and mixed-type data in scalable inference and prediction tasks.

Abstract

Processing high-volume, streaming data is increasingly common in modern statistics and machine learning, where batch-mode algorithms are often impractical because they require repeated passes over the full dataset. This has motivated incremental stochastic estimation methods, including the incremental stochastic Expectation-Maximization (EM) algorithm formulated via stochastic approximation. In this work, we revisit and analyze an incremental stochastic variant of the Majorization-Minimization (MM) algorithm, which generalizes incremental stochastic EM as a special case. Our approach relaxes key EM requirements, such as explicit latent-variable representations, enabling broader applicability and greater algorithmic flexibility. We establish theoretical guarantees for the incremental stochastic MM algorithm, proving consistency in the sense that the iterates converge to a stationary point characterized by a vanishing gradient of the objective. We demonstrate these advantages on a softmax-gated mixture of experts (MoE) regression problem, for which no stochastic EM algorithm is available. Empirically, our method consistently outperforms widely used stochastic optimizers, including stochastic gradient descent, root mean square propagation, adaptive moment estimation, and second-order clipped stochastic optimization. These results support the development of new incremental stochastic algorithms, given the central role of softmax-gated MoE architectures in contemporary deep neural networks for heterogeneous data modeling. Beyond synthetic experiments, we also validate practical effectiveness on two real-world datasets, including a bioinformatics study of dent maize genotypes under drought stress that integrates high-dimensional proteomics with ecophysiological traits, where incremental stochastic MM yields stable gains in predictive performance.

Figures (4)

• Figure 1: Visualization of the synthetic dataset generated from the true parameters and estimated clusters and regression functions by our proposed model with perturbed ground truth initialization in softmax-gated MoE model.
• Figure 2: Maximum log-likelihood function with and without applying the Polyak average method over each iteration and value of selected parameters over $N=1600$ iteration.
• Figure 3: Distance to the true maximum log-likelihood of all algorithms for each iteration.
• Figure 4: Maximum log-likelihood function with and without applying the Polyak averaging method over each iteration for higher dimensional synthetic data.

Theorems & Definitions (28)

• Proposition 1
• Proposition 2
• Theorem 1: Consistency
• Proposition 3
• Theorem 2: Stability and consistency for \ref{['algorithm_onlineMM_MoE']}
• Proposition 4
• Theorem 3: Stability and consistency for \ref{['algorithm_onlineMM_SGMLMoE']}
• Remark 1
• Lemma 1
• proof