Clustering performance analysis using a new correlation-based cluster validity index
Nathakhun Wiroonsri
TL;DR
The paper tackles the challenge of selecting the number of clusters when ground truth is unavailable by introducing a correlation-based validity index for crisp clustering, $NC$, and a multi-peak extension, $NCI$, that yields several plausible candidate cluster counts. $NC(k)$ measures the linear association between pairwise data-point distances and the distances between their cluster centroids, while $NCI$ combines successive improvements to produce multiple local peaks for ranking options. Extensive experiments on artificial benchmarks, real-world UCI datasets, and an online retail dataset demonstrate that $NC$/$NCI$ produce multiple meaningful peaks and often rank favorably against eleven established indices. An R package, NCvalid, is provided to enable practical use and comparison with other indices in marketing analytics and pattern recognition tasks.
Abstract
There are various cluster validity indices used for evaluating clustering results. One of the main objectives of using these indices is to seek the optimal unknown number of clusters. Some indices work well for clusters with different densities, sizes, and shapes. Yet, one shared weakness of those validity indices is that they often provide only one optimal number of clusters. That number is unknown in real-world problems, and there might be more than one possible option. We develop a new cluster validity index based on a correlation between an actual distance between a pair of data points and a centroid distance of clusters that the two points occupy. Our proposed index constantly yields several local peaks and overcomes the previously stated weakness. Several experiments in different scenarios, including UCI real-world data sets, have been conducted to compare the proposed validity index with several well-known ones. An R package related to this new index called NCvalid is available at https://github.com/nwiroonsri/NCvalid.