Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One-Pass Sparsified Gaussian Mixtures

Eric Kightley, Stephen Becker

TL;DR

SGMM tackles clustering in high-dimensional, streaming settings by compressing each data point to a $Q$-dimensional sketch using ROS-based preconditioning and per-point projections. The EM framework is adapted to a sparsified Gaussian density, enabling $\mathcal{O}(K N Q)$ ML updates while estimating parameters in the full $P$-space. Empirical results on MNIST and synthetic data demonstrate substantial speedups with minimal accuracy loss and robust small-cluster and mean recovery under sparsification. This approach is well suited for streaming, resource-constrained clustering in large-scale, high-dimensional applications.

Abstract

We present a one-pass sparsified Gaussian mixture model (SGMM). Given $N$ data points in $P$ dimensions, $X$, the model fits $K$ Gaussian distributions to $X$ and (softly) classifies each point to these clusters. After paying an up-front cost of $\mathcal{O}(NP\log P)$ to precondition the data, we subsample $Q$ entries of each data point and discard the full $P$-dimensional data. SGMM operates in $\mathcal{O}(KNQ)$ time per iteration for diagonal or spherical covariances, independent of $P$, while estimating the model parameters in the full $P$-dimensional space, making it one-pass and hence suitable for streaming data. We derive the maximum likelihood estimators for the parameters in the sparsified regime, demonstrate clustering on synthetic and real data, and show that SGMM is faster than GMM while preserving accuracy.

One-Pass Sparsified Gaussian Mixtures

TL;DR

SGMM tackles clustering in high-dimensional, streaming settings by compressing each data point to a -dimensional sketch using ROS-based preconditioning and per-point projections. The EM framework is adapted to a sparsified Gaussian density, enabling ML updates while estimating parameters in the full -space. Empirical results on MNIST and synthetic data demonstrate substantial speedups with minimal accuracy loss and robust small-cluster and mean recovery under sparsification. This approach is well suited for streaming, resource-constrained clustering in large-scale, high-dimensional applications.

Abstract

We present a one-pass sparsified Gaussian mixture model (SGMM). Given data points in dimensions, , the model fits Gaussian distributions to and (softly) classifies each point to these clusters. After paying an up-front cost of to precondition the data, we subsample entries of each data point and discard the full -dimensional data. SGMM operates in time per iteration for diagonal or spherical covariances, independent of , while estimating the model parameters in the full -dimensional space, making it one-pass and hence suitable for streaming data. We derive the maximum likelihood estimators for the parameters in the sparsified regime, demonstrate clustering on synthetic and real data, and show that SGMM is faster than GMM while preserving accuracy.

Paper Structure

This paper contains 12 sections, 2 theorems, 28 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Theory
  3. Data Sketching
  4. Mixture Models
  5. The EM Algorithm
  6. EM for Sparsified Gaussian Mixtures
  7. Algorithm and Implementation
  8. Simulations
  9. Accuracy and Timing
  10. Small Cluster Recovery
  11. One-Pass Recovery of Means
  12. Conclusions

Key Result

Theorem 1

The maximum likelihood estimator for $\pi_k$ with respect to $Q^\mathcal{R}$ is The maximum likelihood estimators for $\bm{\mu}_k$ and $\mathbf{S}_k$ are the solutions to the system where is the scatter matrix.

Figures (4)

  • Figure 1: Error in $p_k^\mathcal{R}$ as a function of compression. 10000 $\mathbf{x}_i \sim \mathcal{N}(0,1)$ in 100 dimensions per trial. Inset: error in ${D_{{\boldsymbol{\theta}}_k}^\mathcal{R}(\mathbf{x}_i)}$.
  • Figure 2: Accuracy and timing of diagonal SGMM on the subset {0,3,9} of MNIST ($N = 18003$) as a function of compression. Three initializations per trial, 20 trials per compression. Shaded regions indicate standard deviation (dark) and extrema (light) taken over the trials.
  • Figure 3: Small cluster recovery using spherical SGMM.
  • Figure 4: Dependence of one-pass mean estimates on the number of shared features $Q_S$ in the sparsification. SGMM run on the subset $\{0,3,9\}$ of MNIST with $Q=10$; spherical covariances.

Theorems & Definitions (3)

  • Theorem 1: Maximum Likelihood Estimators for Sparsified Gaussian Mixtures
  • proof
  • Corollary 2: MLEs for diagonal $\mathbf{S}_k$