Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fourier Approach to the Parameter Estimation Problem for One-dimensional Gaussian Mixture Models

Xinyu Liu, Hai Zhang

TL;DR

There exists a fundamental limit to the problem of estimating the number of Gaussian components or model order in the mixture model if the number of i.i.i.d samples is finite, and a lower bound is derived in terms of the number of i.i.d samples, the variance, and the number of Gaussian components.

Abstract

The purpose of this paper is twofold. First, we propose a novel algorithm for estimating parameters in one-dimensional Gaussian mixture models (GMMs). The algorithm takes advantage of the Hankel structure inherent in the Fourier data obtained from independent and identically distributed (i.i.d) samples of the mixture. For GMMs with a unified variance, a singular value ratio functional using the Fourier data is introduced and used to resolve the variance and component number simultaneously. The consistency of the estimator is derived. Compared to classic algorithms such as the method of moments and the maximum likelihood method, the proposed algorithm does not require prior knowledge of the number of Gaussian components or good initial guesses. Numerical experiments demonstrate its superior performance in estimation accuracy and computational cost. Second, we reveal that there exists a fundamental limit to the problem of estimating the number of Gaussian components or model order in the mixture model if the number of i.i.d samples is finite. For the case of a single variance, we show that the model order can be successfully estimated only if the minimum separation distance between the component means exceeds a certain threshold value and can fail if below. We derive a lower bound for this threshold value, referred to as the computational resolution limit, in terms of the number of i.i.d samples, the variance, and the number of Gaussian components. Numerical experiments confirm this phase transition phenomenon in estimating the model order. Moreover, we demonstrate that our algorithm achieves better scores in likelihood, AIC, and BIC when compared to the EM algorithm.

A Fourier Approach to the Parameter Estimation Problem for One-dimensional Gaussian Mixture Models

TL;DR

There exists a fundamental limit to the problem of estimating the number of Gaussian components or model order in the mixture model if the number of i.i.i.d samples is finite, and a lower bound is derived in terms of the number of i.i.d samples, the variance, and the number of Gaussian components.

Abstract

The purpose of this paper is twofold. First, we propose a novel algorithm for estimating parameters in one-dimensional Gaussian mixture models (GMMs). The algorithm takes advantage of the Hankel structure inherent in the Fourier data obtained from independent and identically distributed (i.i.d) samples of the mixture. For GMMs with a unified variance, a singular value ratio functional using the Fourier data is introduced and used to resolve the variance and component number simultaneously. The consistency of the estimator is derived. Compared to classic algorithms such as the method of moments and the maximum likelihood method, the proposed algorithm does not require prior knowledge of the number of Gaussian components or good initial guesses. Numerical experiments demonstrate its superior performance in estimation accuracy and computational cost. Second, we reveal that there exists a fundamental limit to the problem of estimating the number of Gaussian components or model order in the mixture model if the number of i.i.d samples is finite. For the case of a single variance, we show that the model order can be successfully estimated only if the minimum separation distance between the component means exceeds a certain threshold value and can fail if below. We derive a lower bound for this threshold value, referred to as the computational resolution limit, in terms of the number of i.i.d samples, the variance, and the number of Gaussian components. Numerical experiments confirm this phase transition phenomenon in estimating the model order. Moreover, we demonstrate that our algorithm achieves better scores in likelihood, AIC, and BIC when compared to the EM algorithm.
Paper Structure (21 sections, 11 theorems, 116 equations, 7 figures, 1 table, 6 algorithms)

This paper contains 21 sections, 11 theorems, 116 equations, 7 figures, 1 table, 6 algorithms.

Table of Contents

  1. Introduction
  2. Mathematical Model and Main Contributions
  3. Paper Organization
  4. Parameter estimation of 1d Gaussian mixtures
  5. Notation and Preliminaries
  6. Estimation of Variance and Model Order
  7. Estimation of Means and Weights
  8. Computation Resolution Limit for Resolving Model Order
  9. Computational Complexity
  10. Comparison with EM Algorithm
  11. Phase Transition in the model order estimation problem
  12. Phase transition
  13. The model order estimation problem
  14. Conclusion and Future Works
  15. Proof of Theorem \ref{['thm:rank']}
  16. ...and 6 more sections

Key Result

Theorem 1

In the noiseless case i.e. $\boldsymbol{W}= \boldsymbol{0}$, assume that Then $u = v, l = k \Longleftrightarrow r(u, l) = + \infty.$

Figures (7)

  • Figure 1: Left: The continuous Fourier data with ten sampled points. The variance is set as $v = 0.450$ with 3 equally weighted components whose means are $(0.3\pi,\pi,1.6\pi)$. Right: The singular value ratio of the Fourier data. The ratio functional with $k=3$ peaks at 0.455.
  • Figure 2: Comparison with the EM algorithm when the model order is known. For each sample size, we conducted 100 trials. The left plot shows the accuracy for variance estimation; The middle one shows the accuracy of the mixing distribution; The right one shows the average running time of each trial.
  • Figure 3: Comparison with EM algorithm for overlapping components. For each sample size, we conducted 100 trials. The left plot shows the accuracy for variance estimation; The middle one shows the accuracy of the mean estimation; The right one shows the average running time of each trial.
  • Figure 4: Comparison with EM algorithm (5 components). For each sample size, we conduct 100 trials. The upper left plot shows the probability density function of the distribution. The upper right one shows the accuracy for variance estimation; The lower left one shows the accuracy of the mixing distribution estimation; The lower right one shows the average running time of each trial.
  • Figure 5: Comparison with the EM algorithm for different separation distances. For each different separation distance, we conducted $100$ trials. The upper left plot shows the accuracy of the variance estimation, the region that Algorithm \ref{['algo: Parameter Estimation Algorithm']} is less accurate than the EM falls in the overlapping region in Figure \ref{['fig:resolution number']}; The upper right plot shows the accuracy of the mixing distribution; The lower left one shows the average running time of each trial; The lower right one shows the average log-likelihood of each estimated model.
  • ...and 2 more figures

Theorems & Definitions (14)

  • Theorem 1
  • Remark 2
  • Theorem 3
  • Remark 4
  • Theorem 5
  • Proposition 6
  • Theorem 7
  • Remark 8
  • Lemma 9
  • Lemma 10
  • ...and 4 more