Measuring the Validity of Clustering Validation Datasets

TL;DR

The paper tackles the problem that using labeled classes as ground truth for clustering validation may misrepresent true cluster structure, undermining external validation. It introduces CLM as a criterion for dataset reliability and proposes Adjusted Internal Validation Measures (IVM$_A$s) that can compare CLM across datasets. By formulating four across-dataset axioms (A1–A4) and generalization protocols (T1–T4), the authors transform six widely used IVMs into IVM$_A$s (notably CH$_A$) that yield consistent, scalable measurements of CLM within and across datasets. Empirical results show that IVM$_A$s correlate more strongly with ground-truth CLM rankings than standard IVMs or classifiers and enable practical applications such as ranking benchmark datasets and improving CLM via data subspace selection. The work provides a principled framework for evaluating and enhancing the quality of clustering benchmarks, with implications for more reliable external clustering validation in diverse domains.

Abstract

Clustering techniques are often validated using benchmark datasets where class labels are used as ground-truth clusters. However, depending on the datasets, class labels may not align with the actual data clusters, and such misalignment hampers accurate validation. Therefore, it is essential to evaluate and compare datasets regarding their cluster-label matching (CLM), i.e., how well their class labels match actual clusters. Internal validation measures (IVMs), like Silhouette, can compare CLM over different labeling of the same dataset, but are not designed to do so across different datasets. We thus introduce Adjusted IVMs as fast and reliable methods to evaluate and compare CLM across datasets. We establish four axioms that require validation measures to be independent of data properties not related to cluster structure (e.g., dimensionality, dataset size). Then, we develop standardized protocols to convert any IVM to satisfy these axioms, and use these protocols to adjust six widely used IVMs. Quantitative experiments (1) verify the necessity and effectiveness of our protocols and (2) show that adjusted IVMs outperform the competitors, including standard IVMs, in accurately evaluating CLM both within and across datasets. We also show that the datasets can be filtered or improved using our method to form more reliable benchmarks for clustering validation.

Figures (7)

• Figure 1: The illustration of how the degree of Cluster-Label Matching (CLM) affects the reliability of external validation. An external validation measure (EVM) evaluates how well the clustering results (markers' shape) match the ground truth partition, typically given by class labels (markers' hue). The CLM is good (A) if the partition formed by the class labels (fill) matches well the clusters formed by the point distribution (encoded by position). If CLM is good (A), the EVM between the dataset labels (A) and clustering results (C, F) gives a reliable evaluation (D, G) of the clustering technique: high/lowEVM (D/G) aligns well with good/badclustering (C/F). But if CLM is bad (B), the EVM is always low (E, H) and unreliable to evaluate whether the clustering is good (C) or bad (F). We aim to evaluate and compare the CLMacross datasets (I,J,K) having diverse characteristics (e.g., dimension, size, data, and class distributions) to inform external validation by distinguishing valid benchmark datasets.
• Figure 2: Results of the ablation study (\ref{['sec:validity']}). Top: data-cardinality test, bottom: dimensionality test. Scatterplots (a-d, f-i) show how the SMAPE varies between IVMs and their variants in average, where each dot corresponds to a single IVM variant $Z_v$. The shape and the color of the dots correspond to $Z$ and $v$, respectively, and $x$ and $y$ coordinates represent the error made by $Z$ (before applying the protocols) and $Z_v$ (after applying the protocols), respectively. If a dot is located in the lower-right half of the scatterplot, it means that the protocol-based variant $Z_v$ produces less error than $Z$. Note that $II$ and $XB$ are horizontally aligned as their adjusted versions are identical. The first three columns show the effect of each protocol and its variants, and the fourth column shows the variant in which all possible protocols are applied ($Z_v = Z_A$). Bar plots (e, j) show the error reduction rate made by the protocols and their combinations averaged over all the IVMs; the higher the bar, the better. Notice that ADJ shows lower error reduction compared to other combinations because not all IVMs benefit from the entire set of protocols. The table below the bar plots indicates what combinations of protocols are applied (O) to each IVM or not (X).
• Figure 3: The cross-validation accuracy of classifiers in classifying classes in 96 labeled datasets. The classifiers are ordered based on the mean accuracy. Error bars indicate 95% confidence interval.
• Figure 4: (a) Distribution of pairwise rank stability for bottom-1/3 (blue; $\mathcal{P}^-$), entire (orange; $\mathcal{P}^*$), and top-1/3 (green; $\mathcal{P}^+$) subsets of 96 labeled datasets based on the CLM score computed by $CH_A$. (b) Rankings of clustering techniques for each set; rankings are not stable and can change dramatically if we use low-quality datasets ((a) blue and orange bars). All rankings are based on ami averaged over each subset. Using the top-ranked datasets leads to more stable and reliable rankings ((a) green bar).
• Figure 5: The runtime of the IVM (blue), IVM$_{A}$ (green), classifiers (red), and the clustering ensemble (Ens.; yellow) in computing the CLM of 96 datasets. The numbers next to each box depict the median runtime of the corresponding measure (left) and the relative time compared to $\{II, XB\}_{A}$, the fastest IVM$_{A}$ (e.g., $CH_A$ is 1.07 times slower than $\{II, XB\}_{A}$).