Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Imbalanced Data Clustering using Equilibrium K-Means

Yudong He

TL;DR

Imbalanced data clustering suffers from a uniform effect where large clusters bias centroid placement. The authors introduce equilibrium K-means (EKM), a Boltzmann-operator–driven objective that imposes centroid repulsion, with weights $w_{kn}= rac{e^{-oldsymbol{ extalpha} d_{kn}}}{ extstyle extstyle\sum_i e^{-oldsymbol{ extalpha} d_{in}}}$ and objective $J_B= abla$ over data, leading to a simple two-step update and batch-learning variant. They unify HKM, FKM, MEFC, and EKM under a gradient-descent SKM framework, analyze convergence conditions, and demonstrate empirically that EKM outperforms baselines on imbalanced data while remaining competitive on balanced data; they also show EKM improves deep clustering representations on MNIST. The practical impact is robust clustering under data imbalance, with scalable performance and the potential to enhance deep clustering pipelines via a more discriminative, repulsion-informed centroid placement.

Abstract

Centroid-based clustering algorithms, such as hard K-means (HKM) and fuzzy K-means (FKM), have suffered from learning bias towards large clusters. Their centroids tend to be crowded in large clusters, compromising performance when the true underlying data groups vary in size (i.e., imbalanced data). To address this, we propose a new clustering objective function based on the Boltzmann operator, which introduces a novel centroid repulsion mechanism, where data points surrounding the centroids repel other centroids. Larger clusters repel more, effectively mitigating the issue of large cluster learning bias. The proposed new algorithm, called equilibrium K-means (EKM), is simple, alternating between two steps; resource-saving, with the same time and space complexity as FKM; and scalable to large datasets via batch learning. We substantially evaluate the performance of EKM on synthetic and real-world datasets. The results show that EKM performs competitively on balanced data and significantly outperforms benchmark algorithms on imbalanced data. Deep clustering experiments demonstrate that EKM is a better alternative to HKM and FKM on imbalanced data as more discriminative representation can be obtained. Additionally, we reformulate HKM, FKM, and EKM in a general form of gradient descent and demonstrate how this general form facilitates a uniform study of K-means algorithms.

Imbalanced Data Clustering using Equilibrium K-Means

TL;DR

Imbalanced data clustering suffers from a uniform effect where large clusters bias centroid placement. The authors introduce equilibrium K-means (EKM), a Boltzmann-operator–driven objective that imposes centroid repulsion, with weights and objective over data, leading to a simple two-step update and batch-learning variant. They unify HKM, FKM, MEFC, and EKM under a gradient-descent SKM framework, analyze convergence conditions, and demonstrate empirically that EKM outperforms baselines on imbalanced data while remaining competitive on balanced data; they also show EKM improves deep clustering representations on MNIST. The practical impact is robust clustering under data imbalance, with scalable performance and the potential to enhance deep clustering pipelines via a more discriminative, repulsion-informed centroid placement.

Abstract

Centroid-based clustering algorithms, such as hard K-means (HKM) and fuzzy K-means (FKM), have suffered from learning bias towards large clusters. Their centroids tend to be crowded in large clusters, compromising performance when the true underlying data groups vary in size (i.e., imbalanced data). To address this, we propose a new clustering objective function based on the Boltzmann operator, which introduces a novel centroid repulsion mechanism, where data points surrounding the centroids repel other centroids. Larger clusters repel more, effectively mitigating the issue of large cluster learning bias. The proposed new algorithm, called equilibrium K-means (EKM), is simple, alternating between two steps; resource-saving, with the same time and space complexity as FKM; and scalable to large datasets via batch learning. We substantially evaluate the performance of EKM on synthetic and real-world datasets. The results show that EKM performs competitively on balanced data and significantly outperforms benchmark algorithms on imbalanced data. Deep clustering experiments demonstrate that EKM is a better alternative to HKM and FKM on imbalanced data as more discriminative representation can be obtained. Additionally, we reformulate HKM, FKM, and EKM in a general form of gradient descent and demonstrate how this general form facilitates a uniform study of K-means algorithms.
Paper Structure (38 sections, 1 theorem, 48 equations, 9 figures, 7 tables, 1 algorithm)

This paper contains 38 sections, 1 theorem, 48 equations, 9 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Existing Efforts to Overcome Uniform Effect
  3. Our Contributions
  4. Organization
  5. K-Means and Its Variations
  6. The Hard K-Means Algorithm (Lloyd's Algorithm)
  7. The Uniform Effect of K-Means
  8. The Fuzzy K-Means Algorithm (Bezdek's Algorithm)
  9. Maximum-Entropy Fuzzy Clustering
  10. Smooth K-Means - A Unified Framework
  11. Objective of Smooth K-Means
  12. The Relationship Between SKM and GMM
  13. Three Common Smooth Minimum Functions
  14. Clustering by Minimizing Smoothed WCSS
  15. The relationship between LogSumExp and Maximum-Entropy Fuzzy Clustering
  16. ...and 23 more sections

Key Result

Theorem 1

The centroid sequence obtained by smooth_km_sgd converges to a (local) minimizer or saddle point of the objective function $J$ if the following conditions can be satisfied:

Figures (9)

  • Figure 1: Clustering results of a highly imbalanced dataset. (a) Ground truth. The colors represent the reference labels of the data points. (b) Clustering by hard K-means. (c) Clustering by fuzzy K-means. (d) Clustering by the proposed equilibrium K-means. They are all two-step alternating algorithms that iteratively update centroids.
  • Figure 2: Objectives $J({\mathbf{c}_1=\mu_1,\mathbf{c}_2})$ as a function of $\mathbf{c}_2$ (the position of the second centroid on the $x$-axis). The yellow, purple, green, and blue curves are reformulated objective functions of HKM, FKM, MEFC, and EKM, respectively. (a), (b), (c), and (d) present objective functions under different data distributions.
  • Figure 3: Gradients as a function of a data point moving along the $x$-axis. (a) Gradients of WCSS, $J_{\text{PN}}$, $J_{\text{LSE}}$, and $J_{\text{B}}$ with fixed smoothing parameters. (b), (c), and (d) are gradients of $J_{\text{PN}}$, $J_{\text{LSE}}$, and $J_{\text{B}}$ with varied smoothing parameters, respectively.
  • Figure 4: Scatter diagrams of artificial datasets. (a)-(d) Primitive scatter diagrams with reference class labels (indicated by colors). (e)-(f) Clustering results of some benchmark algorithms. (i)-(l) Clustering results of the proposed EKM. The black crosses represent centroids obtained by each algorithm.
  • Figure 5: NMI of EKM with different $\alpha$ values on the four artificial datasets. The blue curve is the NMI of EKM, and the red line is the NMI of HKM as a reference.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 1: Convergence Condition
  • proof