Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

K*-Means: A Parameter-free Clustering Algorithm

Louis Mahon, Mirella Lapata

TL;DR

k$^*$means delivers a parameter-free clustering framework that automatically determines the optimal number of clusters $k^*$ by optimizing a minimum description length (MDL) objective while performing standard $k$-means updates. It extends Lloyd’s algorithm with per-cluster substructures and two mechanisms, maybe-split and maybe-merge, to adapt $k$ during training, and it provides a formal convergence proof. Empirically, it outperforms other methods that do not require a pre-specified $k$ in synthetic scenarios and across diverse real datasets, achieving competitive or near-oracle clustering accuracy while offering faster-than-many-parameter-tuned baselines. The approach scales well with data size and offers a principled, information-theoretic criterion for model selection that avoids manual parameter tuning, making it particularly suitable when the true number of clusters is unknown.

Abstract

Clustering is a widely used and powerful machine learning technique, but its effectiveness is often limited by the need to specify the number of clusters, k, or by relying on thresholds that implicitly determine k. We introduce k*-means, a novel clustering algorithm that eliminates the need to set k or any other parameters. Instead, it uses the minimum description length principle to automatically determine the optimal number of clusters, k*, by splitting and merging clusters while also optimising the standard k-means objective. We prove that k*-means is guaranteed to converge and demonstrate experimentally that it significantly outperforms existing methods in scenarios where k is unknown. We also show that it is accurate in estimating k, and that empirically its runtime is competitive with existing methods, and scales well with dataset size.

K*-Means: A Parameter-free Clustering Algorithm

TL;DR

kmeans delivers a parameter-free clustering framework that automatically determines the optimal number of clusters by optimizing a minimum description length (MDL) objective while performing standard -means updates. It extends Lloyd’s algorithm with per-cluster substructures and two mechanisms, maybe-split and maybe-merge, to adapt during training, and it provides a formal convergence proof. Empirically, it outperforms other methods that do not require a pre-specified in synthetic scenarios and across diverse real datasets, achieving competitive or near-oracle clustering accuracy while offering faster-than-many-parameter-tuned baselines. The approach scales well with data size and offers a principled, information-theoretic criterion for model selection that avoids manual parameter tuning, making it particularly suitable when the true number of clusters is unknown.

Abstract

Clustering is a widely used and powerful machine learning technique, but its effectiveness is often limited by the need to specify the number of clusters, k, or by relying on thresholds that implicitly determine k. We introduce k*-means, a novel clustering algorithm that eliminates the need to set k or any other parameters. Instead, it uses the minimum description length principle to automatically determine the optimal number of clusters, k*, by splitting and merging clusters while also optimising the standard k-means objective. We prove that k*-means is guaranteed to converge and demonstrate experimentally that it significantly outperforms existing methods in scenarios where k is unknown. We also show that it is accurate in estimating k, and that empirically its runtime is competitive with existing methods, and scales well with dataset size.
Paper Structure (18 sections, 3 theorems, 5 equations, 5 figures, 4 tables, 3 algorithms)

This paper contains 18 sections, 3 theorems, 5 equations, 5 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. The k$^*$means Algorithm
  4. Quantifying Description Length
  5. Minimising Description Length
  6. Maybe-Split Step
  7. Maybe-Merge Step
  8. Proof of Convergence
  9. Experimental Evaluation
  10. Synthetic Data
  11. Labelled Datasets
  12. Comparison to Sweeping $k$
  13. Runtime Analysis
  14. Limitations
  15. Conclusion
  16. ...and 3 more sections

Key Result

Lemma 1

At each assign step (which is the same step as vanilla $k$-means), the mdl cost either decreases or remains the same, and it remains the same only if no points are reassigned.

Figures (5)

  • Figure 1: Synthetic data of standard, multivariate Normal clusters, with varying degrees of separation. Left: weak separation, inter-centroid distance constrained to $\geq 2$, k$^*$means is 9% accurate in inferring $k$ and baselines are $\leq$4.4%. Middle: inter-centroid distance constrained to $\geq~3$, k$^*$means is 25% accurate in inferring $k$ and baselines are $\leq$16%. Right: strong separation, inter-centroid distance constrained to $\geq 5$, k$^*$means is 99% accurate in inferring $k$ and baselines are $\leq$57%.
  • Figure 2: Clusters predicted by k$^*$means for the UMAP representations on the Speech Commands dataset, by k$^*$means (left) and XMeans (right). k$^*$means predicts 33 classes and XMeans predicts 315, vs. 36 in the annotations.
  • Figure 3: Runtime as a function of dataset size. Each point represents the mean runtime of 10 randomly sampled subsets from the Speech Commands dataset of the given size. The line tracks windowed averages. k$^*$means has a similar speed to xmeans, faster than HDBSCAN, especially for larger input sizes, and slower than kmeans, DBSCAN and GMM.
  • Figure 4: Values of $k$ (number of clusters) found on MNIST for different values of the DBSCAN parameters, min-pts (x-axis) and eps (y-axis). We sweep min-pts from 1--40, and eps from 0.03 to 1.5 in 5% increments.
  • Figure : Maybe-Split Procedure

Theorems & Definitions (7)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 3
  • proof
  • Remark 4