CoHiRF: A Scalable and Interpretable Clustering Framework for High-Dimensional Data

TL;DR

CoHiRF tackles high-dimensional clustering by integrating random feature sub-sampling, repeated low-dimensional K-Means steps, and a unanimous consensus to form a hierarchical, medoid-based clustering without requiring the true number of clusters $C^*$. The method achieves strong scalability and interpretability, reconstructing final clusters by tracing cluster evolution across iterative steps, and it remains competitive with, and often faster than, state-of-the-art high-dimensional clustering approaches. Empirical results on synthetic and real-world OpenML datasets demonstrate robust performance and significant speed advantages, making CoHiRF a practical tool for large-scale, high-dimensional data analysis. The framework also supports out-of-core computation and allows for hyperparameter tuning to balance accuracy and efficiency, enabling domain experts to explore multi-level clustering structures through interpretable dendrograms.

Abstract

Clustering high-dimensional data poses significant challenges due to the curse of dimensionality, scalability issues, and the presence of noisy and irrelevant features. We propose Consensus Hierarchical Random Feature (CoHiRF), a novel clustering method designed to address these challenges effectively. CoHiRF leverages random feature selection to mitigate noise and dimensionality effects, repeatedly applies K-Means clustering in reduced feature spaces, and combines results through a unanimous consensus criterion. This iterative approach constructs a cluster assignment matrix, where each row records the cluster assignments of a sample across repetitions, enabling the identification of stable clusters by comparing identical rows. Clusters are organized hierarchically, enabling the interpretation of the hierarchy to gain insights into the dataset. CoHiRF is computationally efficient with a running time comparable to K-Means, scalable to massive datasets, and exhibits robust performance against state-of-the-art methods such as SC-SRGF, HDBSCAN, and OPTICS. Experimental results on synthetic and real-world datasets confirm the method's ability to reveal meaningful patterns while maintaining scalability, making it a powerful tool for high-dimensional data analysis.

Figures (8)

• Figure 1: Illustration of one iteration of the CoHiRF algorithm with $q=2$, $C=2$ and $R=4$. At the beginning of step $e$, we start with $n^{(e-1)}$ samples (the medoids inherited from the previous iteration). For each repetition $r\in [R]$, subsample $q$ features at random and cluster the $n^{(e-1)}$ samples using K-Means with $C=2$, then Identify clusters by grouping together the samples consistently clustered together across all repetitions. Obtain that way $n^{(e)}$ newly formed clusters and finally choose a medoid for each of the formed clusters.
• Figure 2: Concept representation for the hierarchical clustering structure built by CoHiRF . The hierarchy is built iteratively by our method without requiring solving any additional expensive optimization problem. The final step of CoHiRF is thus immediate.
• Figure 3: Running time of the tested algorithms for 14427 samples. Because of memory constraints HDBSCAN could not run with more than 4163 features.
• Figure 4: Running time of the tested algorithms for 14427 features. Because of memory constraints and/or time constraints HDBSCAN could not run with more than 4163 features and OPTICS, Affinity Propagation and SC-SRGF with more than 14427 features.
• Figure 5: Running time and ARI for tested algorithm for different inter-cluster distances in an experiment with 1000 samples and 10000 features.