Binary Bleed: Fast Distributed and Parallel Method for Automatic Model Selection

TL;DR

This paper tackles the costly problem of selecting the hyper-parameter $k$ in clustering and dimensionality reduction by introducing Binary Bleed, a threshold-driven binary search that prunes large portions of the $k$-space $\mathcal{K}$ while evaluating a score function $f(k,\mathcal{D})$. By exploiting score distributions that resemble a square-wave around the optimal $k$, Binary Bleed reduces the number of $k$ values evaluated from $O(n)$ toward $O(\log n)$ in practice, while offering Vanilla and Early Stop variants and extending to multi-threaded and distributed HPC environments. The authors demonstrate substantial reductions in visited $k$ and meaningful speedups across NMFk, K-means, and RESCAL—using silhouette scores for maximization and Davies-Bouldin scores for minimization—on synthetic and large-scale real datasets, with implementations and results available on GitHub. Overall, Binary Bleed provides a scalable, robust framework for automatic model selection that can significantly accelerate hyper-parameter tuning in large-scale unsupervised learning workflows. $k$-space pruning and early stopping enable fast, accurate automatic model determination without sacrificing correctness in practical settings.

Abstract

In several Machine Learning (ML) clustering and dimensionality reduction approaches, such as non-negative matrix factorization (NMF), RESCAL, and K-Means clustering, users must select a hyper-parameter k to define the number of clusters or components that yield an ideal separation of samples or clean clusters. This selection, while difficult, is crucial to avoid overfitting or underfitting the data. Several ML applications use scoring methods (e.g., Silhouette and Davies Boulding scores) to evaluate the cluster pattern stability for a specific k. The score is calculated for different trials over a range of k, and the ideal k is heuristically selected as the value before the model starts overfitting, indicated by a drop or increase in the score resembling an elbow curve plot. While the grid-search method can be used to accurately find a good k value, visiting a range of k can become time-consuming and computationally resource-intensive. In this paper, we introduce the Binary Bleed method based on binary search, which significantly reduces the k search space for these grid-search ML algorithms by truncating the target k values from the search space using a heuristic with thresholding over the scores. Binary Bleed is designed to work with single-node serial, single-node multi-processing, and distributed computing resources. In our experiments, we demonstrate the reduced search space gain over a naive sequential search of the ideal k and the accuracy of the Binary Bleed in identifying the correct k for NMFk, K-Means pyDNMFk, and pyDRESCALk with Silhouette and Davies Boulding scores. We make our implementation of Binary Bleed for the NMF algorithm available on GitHub.

Figures (15)

• Figure 1: Traversal ordering sort of pre-order (red), in-order (green), and post-order (blue).
• Figure 2: Vanilla: Part 1
• Figure 3: Vanilla: Part 2
• Figure 4: Binary Bleed Vanilla.
• Figure 5: Early Stop: Part 1