Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Selection and Parameter Estimation of Multi-dimensional Gaussian Mixture Model

Xinyu Liu, Hai Zhang

Abstract

In this paper, we study the problem of learning multi-dimensional Gaussian Mixture Models (GMMs), with a specific focus on model order selection and efficient mixing distribution estimation. We first establish an information-theoretic lower bound on the critical sample complexity required for reliable model selection. More specifically, we show that distinguishing a $k$-component mixture from a simpler model necessitates a sample size scaling of $Ω(Δ^{-(4k-4)})$. We then propose a thresholding-based estimation algorithm that evaluates the spectral gap of an empirical covariance matrix constructed from random Fourier measurement vectors. This parameter-free estimator operates with an efficient time complexity of $\mathcal{O}(k^2 n)$, scaling linearly with the sample size. We demonstrate that the sample complexity of our method matches the established lower bound, confirming its minimax optimality with respect to the component separation distance $Δ$. Conditioned on the estimated model order, we subsequently introduce a gradient-based minimization method for parameter estimation. To effectively navigate the non-convex objective landscape, we employ a data-driven, score-based initialization strategy that guarantees rapid convergence. We prove that this method achieves the optimal parametric convergence rate of $\mathcal{O}_p(n^{-1/2})$ for estimating the component means. To enhance the algorithm's efficiency in high-dimensional regimes where the ambient dimension exceeds the number of mixture components (i.e., \(d > k\)), we integrate principal component analysis (PCA) for dimension reduction. Numerical experiments demonstrate that our Fourier-based algorithmic framework outperforms conventional Expectation-Maximization (EM) methods in both estimation accuracy and computational time.

Model Selection and Parameter Estimation of Multi-dimensional Gaussian Mixture Model

Abstract

In this paper, we study the problem of learning multi-dimensional Gaussian Mixture Models (GMMs), with a specific focus on model order selection and efficient mixing distribution estimation. We first establish an information-theoretic lower bound on the critical sample complexity required for reliable model selection. More specifically, we show that distinguishing a -component mixture from a simpler model necessitates a sample size scaling of . We then propose a thresholding-based estimation algorithm that evaluates the spectral gap of an empirical covariance matrix constructed from random Fourier measurement vectors. This parameter-free estimator operates with an efficient time complexity of , scaling linearly with the sample size. We demonstrate that the sample complexity of our method matches the established lower bound, confirming its minimax optimality with respect to the component separation distance . Conditioned on the estimated model order, we subsequently introduce a gradient-based minimization method for parameter estimation. To effectively navigate the non-convex objective landscape, we employ a data-driven, score-based initialization strategy that guarantees rapid convergence. We prove that this method achieves the optimal parametric convergence rate of for estimating the component means. To enhance the algorithm's efficiency in high-dimensional regimes where the ambient dimension exceeds the number of mixture components (i.e., ), we integrate principal component analysis (PCA) for dimension reduction. Numerical experiments demonstrate that our Fourier-based algorithmic framework outperforms conventional Expectation-Maximization (EM) methods in both estimation accuracy and computational time.
Paper Structure (29 sections, 14 theorems, 144 equations, 6 figures, 3 algorithms)

This paper contains 29 sections, 14 theorems, 144 equations, 6 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Our Contributions
  4. Paper Organization and Notations
  5. Model Selection and Sampling Complexity
  6. Indistinguishable Probability Models
  7. Critical Sample Complexity for Model Selection
  8. A Lower Bound via Moment-Matching
  9. Fourier Measurements and Latent Fourier Covariance Matrix
  10. Signal Strength Analysis
  11. Algorithms and Theoretical Results
  12. Mixing Distribution Estimation
  13. Case 1: Learn GMMs when $k \ge d$
  14. Case 2: Learn GMMs When $k < d$
  15. Numerical Experiments
  16. ...and 14 more sections

Key Result

Theorem 2.1

(Theorem 2.4 in liu2024fourier) Consider the true $k$-component GMM $P$ constructed above with separation distance $\Delta$. For any $0 < \delta < 1/2$, if the sample size $n$ satisfies: where $C_{k,w}$ is a constant depending only on $k$ and the weights, then there exists a $(k-1)$-component GMM $Q$ such that $\mathbb{P}_{X_i \sim P} \left( \frac{1}{n}\sum_{j=1}^n \log p(X_i) \le \frac{1}{n}\sum

Figures (6)

  • Figure 1: Illustration of the Algorithm \ref{['algo: mean estimation 1']}. Left: 500 samples (in blue) draw from a 3-component mixture model with centers $(3.94, 0.72), (-0.12, 4.00)$ and $(-2.91,2.75)$ (in red); Middle: the first $25$ starting point (in green) with the largest score $s(x)$; Right: the updated points after 5 steps of (\ref{['eqn:gradient descent']}) with $\gamma=0.5$ (in green).
  • Figure 2: Geometry of the Gaussian means. The black dots stand for the mean locations and $\Delta$ stands for the separation distance.
  • Figure 3: Empirical success rate of $96$ runs of trials under each $(\log(n), \Delta)$ setting.
  • Figure 4: $\sum_{i=1}^k w_i \mathcal{N}(\mu_i, I_k)$ with $\mu_i$'s drawn uniformly from the sphere with radius $4$. For each figure, the upper plot shows the accuracy of the mixing distribution estimation under the 1-Wasserstein distance, and the lower plot shows the average running time of each trial.
  • Figure 5: (a) Means drawn from sphere with radius $0.1$. (b) Means completely overlap i.e. $\Delta = 0$.
  • ...and 1 more figures

Theorems & Definitions (20)

  • Theorem 2.1
  • Proposition 2.2
  • Theorem 2.3
  • Remark 2.4
  • Lemma 2.5
  • Proposition 2.6: Lower Bound with Random Directional Sampling
  • Theorem 2.7
  • Remark 2.8
  • Theorem 2.9
  • Remark 2.10
  • ...and 10 more