Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Clustering: Large Scale Unsupervised Learning of Gaussian Mixture Models with Outliers

Yijia Zhou, Kyle A. Gallivan, Adrian Barbu

TL;DR

A provably robust clustering algorithm based on loss minimization that performs well on Gaussian mixture models with outliers is introduced that provides theoretical guarantees that the algorithm obtains high accuracy with high probability under certain assumptions.

Abstract

Clustering is a widely used technique with a long and rich history in a variety of areas. However, most existing algorithms do not scale well to large datasets, or are missing theoretical guarantees of convergence. This paper introduces a provably robust clustering algorithm based on loss minimization that performs well on Gaussian mixture models with outliers. It provides theoretical guarantees that the algorithm obtains high accuracy with high probability under certain assumptions. Moreover, it can also be used as an initialization strategy for $k$-means clustering. Experiments on real-world large-scale datasets demonstrate the effectiveness of the algorithm when clustering a large number of clusters, and a $k$-means algorithm initialized by the algorithm outperforms many of the classic clustering methods in both speed and accuracy, while scaling well to large datasets such as ImageNet.

Scalable Clustering: Large Scale Unsupervised Learning of Gaussian Mixture Models with Outliers

TL;DR

A provably robust clustering algorithm based on loss minimization that performs well on Gaussian mixture models with outliers is introduced that provides theoretical guarantees that the algorithm obtains high accuracy with high probability under certain assumptions.

Abstract

Clustering is a widely used technique with a long and rich history in a variety of areas. However, most existing algorithms do not scale well to large datasets, or are missing theoretical guarantees of convergence. This paper introduces a provably robust clustering algorithm based on loss minimization that performs well on Gaussian mixture models with outliers. It provides theoretical guarantees that the algorithm obtains high accuracy with high probability under certain assumptions. Moreover, it can also be used as an initialization strategy for -means clustering. Experiments on real-world large-scale datasets demonstrate the effectiveness of the algorithm when clustering a large number of clusters, and a -means algorithm initialized by the algorithm outperforms many of the classic clustering methods in both speed and accuracy, while scaling well to large datasets such as ImageNet.
Paper Structure (17 sections, 14 theorems, 84 equations, 10 figures, 2 tables, 1 algorithm)

This paper contains 17 sections, 14 theorems, 84 equations, 10 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Reviews
  3. Scalable Clustering by Robust Loss Minimization
  4. Robust Loss Function
  5. Theoretical Guarantees
  6. Computational Complexity
  7. Experiments
  8. Simulation Experiments
  9. Comparison of Observed and Theoretical Accuracy
  10. Stability of SCRLM w.r.t. the Bandwidth Parameter
  11. Comparison with other clustering methods
  12. Real Data Experiments
  13. Conclusion
  14. Preliminaries
  15. Proof of Key Lemmas and Propositions
  16. ...and 2 more sections

Key Result

Theorem 1

Given $N$ samples from a GMM with outliers, with $w_i\geq a/m, i=\overline{1,m}$ for some $a>0$ and $\sigma_{max}\leq \rho < \sqrt{0.6}$, then Algorithm alg:scRLM (SCLRM) using $|S|=n$ subsamples has $100\%$ accuracy with probability at least

Figures (10)

  • Figure 1: Structure of the Gaussian mixture model with outliers used in this paper.
  • Figure 2: The robust loss function $\ell(\mathbf{d}; \rho)$ for different values of $p$ and $\rho$.
  • Figure 3: Diagram illustrating Algorithm \ref{['alg:scRLM']}.
  • Figure 4: Comparison between the parameter combinations where the SCRLM algorithm is theoretically guaranteed to have 100% accuracy for 99% of the time with the experimental findings.
  • Figure 5: Evaluation of tightness of the bandwidth parameter $\rho$.
  • ...and 5 more figures

Theorems & Definitions (27)

  • Theorem 1
  • Corollary 1
  • Lemma 1
  • Corollary 2
  • proof
  • Corollary 3
  • proof
  • Corollary 4: Separation between negatives
  • proof
  • Corollary 5: Concentration of positives in the same cluster
  • ...and 17 more