Method of moments for Gaussian mixtures: Implementation and benchmarks

TL;DR

The paper investigates parameter recovery for Gaussian mixture models using the method of moments, emphasizing high-dimensional, non-diagonal covariance settings. It provides a practical MOM-based pipeline, including exact and sample moments, and introduces two off-diagonal generation strategies (tensor and recursive) to recover covariances. A Julia implementation, GMMParameterEstimation, is released and benchmarked, showing machine-precision recovery with exact moments and robust performance with sample moments, while highlighting the trade-offs between moment selections and system size. The work demonstrates the viability and scalability of MOM-based parameter recovery for complex Gaussian mixtures and offers practical guidance for moment selection, solution filtering, and algorithmic choices. Overall, the results support MOM as a competitive, assumption-light alternative to likelihood-based methods, with strong implications for fast, interpretable mixture-model parameter recovery in high dimensions.

Abstract

Gaussian mixture models are universal approximators in the sense that any smooth density can be approximated arbitrarily well with a Gaussian mixture model with enough components. Due to their broad expressive power, Gaussian mixture models appear in many applications. As a result, algebraic parameter recovery for Gaussian mixture models from data is a valuable contribution to multiple fields. Our work documents performance of the method of moments for high dimensional Gaussian mixtures. We outline the method of moments, and selections of moments and their corresponding polynomials that work well for parameter recovery in practice. Our main contribution puts these ideas into practice with an implementation as a julia package, GMMParameterEstimation, as well as computational benchmarks.

Figures (7)

• Figure 1: Sample from a 2-dimensional Gaussian 2-mixture model, $n=10000$
• Figure 2: (Sample moments, unknown mixing coefficients, comparison of the two recursive systems) Median mixing, mean, and covariance errors, timings, and pass rates over 1000 random test runs with unknown mixing coefficients normalized by the number of parameters.
• Figure 3: (Exact moments, mixing coefficients known) Median mean, and covariance errors for exact moments over 1000 random test runs with known mixing coefficients normalized by the number of parameters.
• Figure 4: (Exact moments, $\Sigma_\ell\succ0$) Median mixing coefficient, mean, and covariance errors for exact moments over 1000 random test runs with unknown mixing coefficients normalized by the number of parameters.
• Figure 5: (Sample moments, $\Sigma_\ell\succ0$) Median mixing coefficient, mean, and covariance errors, and number of statistically meaningful solutions for sample moments over 1000 random test runs with unknown mixing coefficients with $k = 3$ normalized by the number of parameters.

Theorems & Definitions (6)

• Example 2.1
• Example 2.2
• Remark 2.3
• Example 2.4
• Remark 3.1
• Remark 3.2