Learning Mixture Models via Efficient High-dimensional Sparse Fourier Transforms
Alkis Kalavasis, Pravesh K. Kothari, Shuchen Li, Manolis Zampetakis
TL;DR
The paper tackles the problem of learning parameters of high-dimensional mixture models with heavy-tailed components, introducing a Fourier-based framework that moves beyond the limitations of moment-based methods. It develops a robust high-dimensional sparse Fourier transform (SFT) to recover k frequency components from noisy observations, enabling efficient estimation when component distributions exhibit Slow Fourier Decay (SFD). The main contributions include a two-stage algorithm for mixtures: (i) recover SFD component means without mean separation, and (ii) recover Fast Fourier Decay (FFD) components using Sum-of-Squares (SoS) techniques, with a careful handling of cross-component separations via resilience properties. The results show polynomial-time and sample complexity for SFD mixtures and a polynomial-SOS augmented approach for mixtures containing both SFD and FFD components, along with a strong negative result for moment-based methods and robust mean-estimation guarantees under noise-oblivious contamination. This work provides a versatile toolkit for high-dimensional distribution learning, offering robust performance for heavy-tailed distributions and suggesting broad applicability beyond traditional GMMs, including robust statistics under adversarial settings.
Abstract
In this work, we give a ${\rm poly}(d,k)$ time and sample algorithm for efficiently learning the parameters of a mixture of $k$ spherical distributions in $d$ dimensions. Unlike all previous methods, our techniques apply to heavy-tailed distributions and include examples that do not even have finite covariances. Our method succeeds whenever the cluster distributions have a characteristic function with sufficiently heavy tails. Such distributions include the Laplace distribution but crucially exclude Gaussians. All previous methods for learning mixture models relied implicitly or explicitly on the low-degree moments. Even for the case of Laplace distributions, we prove that any such algorithm must use super-polynomially many samples. Our method thus adds to the short list of techniques that bypass the limitations of the method of moments. Somewhat surprisingly, our algorithm does not require any minimum separation between the cluster means. This is in stark contrast to spherical Gaussian mixtures where a minimum $\ell_2$-separation is provably necessary even information-theoretically [Regev and Vijayaraghavan '17]. Our methods compose well with existing techniques and allow obtaining ''best of both worlds" guarantees for mixtures where every component either has a heavy-tailed characteristic function or has a sub-Gaussian tail with a light-tailed characteristic function. Our algorithm is based on a new approach to learning mixture models via efficient high-dimensional sparse Fourier transforms. We believe that this method will find more applications to statistical estimation. As an example, we give an algorithm for consistent robust mean estimation against noise-oblivious adversaries, a model practically motivated by the literature on multiple hypothesis testing. It was formally proposed in a recent Master's thesis by one of the authors, and has already inspired follow-up works.