Convergence and Optimality of the EM Algorithm Under Multi-Component Gaussian Mixture Models

TL;DR

The population-level EM is shown to converge to the true parameter when the smallest separation among all pairs of Gaussian components exceeds a logarithmic factor of the largest separation and the reciprocal of the minimal mixing probabilities, and at the sample level, the EM algorithm is shown to be minimax rate-optimal.

Abstract

Gaussian mixture models (GMMs) are fundamental statistical tools for modeling heterogeneous data. Due to the nonconcavity of the likelihood function, the Expectation-Maximization (EM) algorithm is widely used for parameter estimation of each Gaussian component. Existing analyses of the EM algorithm's convergence to the true parameter focus on either the two-component case or multi-component settings with known mixing probabilities and isotropic covariance matrices. In this work, we study the convergence of the EM algorithm for multi-component GMMs in full generality. The population-level EM is shown to converge to the true parameter when the smallest separation among all pairs of Gaussian components exceeds a logarithmic factor of the largest separation and the reciprocal of the minimal mixing probabilities. At the sample level, the EM algorithm is shown to be minimax rate-optimal, up to a logarithmic factor. We develop two distinct novel analytical approaches, each tailored to a different regime of separation, reflecting two complementary perspectives on the use of EM. As a byproduct of our analysis, we show that the EM algorithm, when used for community detection, also achieves the minimax optimal rate of misclustering error under milder separation conditions than spectral clustering and Lloyd's algorithm, an interesting result in its own right. Our analysis allows the number of components, the minimal mixing probabilities, the separation between Gaussian components and the dimension to grow with the sample size. Simulation studies corroborate our theoretical findings.

Figures (4)

• Figure 1: $\Delta_{\min}$ requirement in which the EM estimation achieves the rates in \ref{['rate_EM_final_known']} and \ref{['rate_EM_final_unknown']}. Left: known $\Sigma^*$; Right: unknown $\Sigma^*$.
• Figure 2: Plots of the iteration number versus log optimization error $\log(d(\widehat{M}^{(t)},\widehat{M}))$ and log statistical error $\log(d(\widehat{M}^{(t)},M^*))$. Left: balanced mixing probability $\pi^*=(1/3,1/3,1/3)$. Right: imbalanced mixing probability $\pi^*=(0.6,0.2,0.2)$.
• Figure 3: Convergence of EM on the mixture of $L=3$ Gaussians in $\mathbb{R}^{10}$. The first row (known $\Sigma^*$) and the second row (unknown $\Sigma^*$) show average mean estimation statistical error under different separations; the bottom row (unknown $\Sigma^*$) shows average covariance estimation statistical error. Left: balanced mixing probability $\pi^*=(1/3,1/3,1/3)$. Right: imbalanced mixing probability $\pi^*=(0.6,0.2,0.2)$.
• Figure 4: Empirical vs. optimal rates. Left: isotropic model; Right: compound symmetry model. Top row: mean; Bottom row: covariance.

Theorems & Definitions (82)

• Lemma 1
• Theorem 2
• proof
• Corollary 3: Known $\Sigma^*$
• Remark 1: Separation requirement
• Remark 2: Initialization
• Corollary 4: A special initialization of $\Sigma^*$
• proof
• Theorem 5
• proof