Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimating Gaussian mixtures using sparse polynomial moment systems

Julia Lindberg, Carlos Améndola, Jose Israel Rodriguez

Abstract

The method of moments is a classical statistical technique for density estimation that solves a system of moment equations to estimate the parameters of an unknown distribution. A fundamental question critical to understanding identifiability asks how many moment equations are needed to get finitely many solutions and how many solutions there are. We answer this question for classes of Gaussian mixture models using the tools of polyhedral geometry. In addition, we show that a generic Gaussian $k$-mixture model is identifiable from its first $3k+2$ moments. Using these results, we present a homotopy algorithm that performs parameter recovery for high dimensional Gaussian mixture models where the number of paths tracked scales linearly in the dimension.

Estimating Gaussian mixtures using sparse polynomial moment systems

Abstract

The method of moments is a classical statistical technique for density estimation that solves a system of moment equations to estimate the parameters of an unknown distribution. A fundamental question critical to understanding identifiability asks how many moment equations are needed to get finitely many solutions and how many solutions there are. We answer this question for classes of Gaussian mixture models using the tools of polyhedral geometry. In addition, we show that a generic Gaussian -mixture model is identifiable from its first moments. Using these results, we present a homotopy algorithm that performs parameter recovery for high dimensional Gaussian mixture models where the number of paths tracked scales linearly in the dimension.

Paper Structure

This paper contains 26 sections, 18 theorems, 89 equations, 4 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Preliminaries
  4. Method of moments
  5. Genericity
  6. Statistically meaningful solutions
  7. Sparse polynomial systems
  8. Mixed volumes
  9. BKK bound
  10. Rational Identifiability for Gaussian mixture models
  11. Parameter recovery in one dimension
  12. Mixed volume of -weighted models
  13. Mixed volume of -weighted homoscedastic models
  14. Finding all solutions using homotopy continuation
  15. Means unknown
  16. ...and 11 more sections

Key Result

Theorem 1.1

Goodfellow-et-al-2016 A Gaussian mixture model is a universal approximator of densities, in the sense that any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components.

Figures (4)

  • Figure 1: Two distinct Gaussian mixture densities with $k = 3$ components and the same first eight moments.
  • Figure 2: Individual components of two Gaussian mixture models with similar mixture densities.
  • Figure 3: The region bounded by the dotted blue curve is the space of realizable moments for this model. The red curve given by $3m_3^4+2(m_4-3)^3 =0$ is the set of non-generic moments. The point $(0,3)$ is labeled by a purple cross is non-generic and corresponds to the third and fourth moment of a univariate Gaussian. The red point $(\frac{1}{5},2)$ has four real solutions to the moment equations while the green point $(\frac{1}{5},4)$ has two real solutions. Any choice of moments in the region bounded by the dotted curve and in the complement of the red curve will yield a set of moment equations with six complex solutions, of which two are statistically meaningful (one up to label swapping symmetry).
  • Figure 4: Region in the space of parameters $\overline{m}_1,\overline{m}_2,\sigma^2$ where there are statistically meaningful solutions for $k=2$ mixture model with unknown means and $\lambda_1 = \lambda_2 = \frac{1}{2}$.

Theorems & Definitions (52)

  • Theorem 1.1
  • Example 1.4
  • Example 2.1
  • Definition 2.2
  • Definition 2.3
  • Remark 2.4
  • Example 2.5
  • Theorem 2.6: Bernstein-Khovanskii-Kouchnirenko Bound
  • Remark 2.7
  • Remark 2.8
  • ...and 42 more