Absolute indices for determining compactness, separability and number of clusters

TL;DR

The paper addresses the problem of identifying the true number of clusters by introducing absolute cluster validity indices based on a cluster-specific compactness function and neighbor-based separability margins. It defines $f(t)$, the $\varepsilon$-compactness index $c_A(\varepsilon)$, and the aggregated $C_k(\varepsilon)$ to measure intra-cluster compactness, while introducing adjacent sets, margins, and neighbor relations to quantify inter-cluster separability via $\hat{s}_k$. The number of clusters is selected through a multiobjective framework using the decision-space plot and the combined index $T_k(\varepsilon) = C_k(\varepsilon) + \hat{s}_k$, favoring high, non-dominated solutions. The approach is validated on synthetic and real-world datasets, showing competitive performance and robustness relative to traditional indices, with the added advantage of scale-invariance and interpretability through the decision-space visualization.

Abstract

Finding "true" clusters in a data set is a challenging problem. Clustering solutions obtained using different models and algorithms do not necessarily provide compact and well-separated clusters or the optimal number of clusters. Cluster validity indices are commonly applied to identify such clusters. Nevertheless, these indices are typically relative, and they are used to compare clustering algorithms or choose the parameters of a clustering algorithm. Moreover, the success of these indices depends on the underlying data structure. This paper introduces novel absolute cluster indices to determine both the compactness and separability of clusters. We define a compactness function for each cluster and a set of neighboring points for cluster pairs. This function is utilized to determine the compactness of each cluster and the whole cluster distribution. The set of neighboring points is used to define the margin between clusters and the overall distribution margin. The proposed compactness and separability indices are applied to identify the true number of clusters. Using a number of synthetic and real-world data sets, we demonstrate the performance of these new indices and compare them with other widely-used cluster validity indices.

Figures (11)

• Figure 1: Compactness functions
• Figure 2: Illustration of sets $V_1(\varepsilon),~V_2(\varepsilon)$ and $\bar{V}_1(\varepsilon)$
• Figure 3: Compactness indices of clusters in the synthetic data
• Figure 4: Dependence of compactness index $c_A$ on $\varepsilon$
• Figure 5: Compactness indices of clusters - D15112

Theorems & Definitions (15)

• Definition 1
• Proposition 1
• proof
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Definition 7
• Proposition 2