Fast $k$-means Seeding Under The Manifold Hypothesis

TL;DR

This work addresses the challenge of clustering with $k$-means under a realistic data-generating model by adopting the manifold hypothesis, which posits that data of ambient dimension $D$ concentrate near a smooth $d$-dimensional manifold with $\varepsilon=2/d$. By combining optimum-quantization ideas with beyond-worst-case analysis, the authors derive finite-sample scaling laws for the distortion and data-dependent quantities $\beta_k$ and $\eta$, and leverage these insights to design $\operatorname{Qkmeans}$, a fast seeding method based on rejection sampling from a simple proposal distribution $\kappa(\cdot|C)$ augmented by ANN acceleration. They establish performance guarantees, including a main result that yields $\mathbb{E}[\operatorname{cost}(\mathcal{X},C)]$ bounds with a $\mathcal{O}(\rho^{-2})$ factor and a decaying additive term, plus a corollary for pure rejection sampling under the manifold hypothesis with $\mathcal{O}(\rho^{-2}\log k)$ seeding guarantees. An extensive empirical study across image, text, and sensor data validates the scaling laws, corroborates the relationship between estimated quantization exponent and intrinsic dimension, and demonstrates 5–10x speedups of $\operatorname{Qkmeans}$ over prior seeding methods while maintaining competitive seeding quality.

Abstract

We study beyond worst case analysis for the $k$-means problem where the goal is to model typical instances of $k$-means arising in practice. Existing theoretical approaches provide guarantees under certain assumptions on the optimal solutions to $k$-means, making them difficult to validate in practice. We propose the manifold hypothesis, where data obtained in ambient dimension $D$ concentrates around a low dimensional manifold of intrinsic dimension $d$, as a reasonable assumption to model real world clustering instances. We identify key geometric properties of datasets which have theoretically predictable scaling laws depending on the quantization exponent $\varepsilon = 2/d$ using techniques from optimum quantization theory. We show how to exploit these regularities to design a fast seeding method called $\operatorname{Qkmeans}$ which provides $O(ρ^{-2} \log k)$ approximate solutions to the $k$-means problem in time $O(nD) + \widetilde{O}(\varepsilon^{1+ρ}ρ^{-1}k^{1+γ})$; where the exponent $γ= \varepsilon + ρ$ for an input parameter $ρ< 1$. This allows us to obtain new runtime - quality tradeoffs. We perform a large scale empirical study across various domains to validate our theoretical predictions and algorithm performance to bridge theory and practice for beyond worst case data clustering.

Figures (11)

• Figure 1: Scaling behavior across datasets
• Figure 2: Dependence of quantization exponent on ID
• Figure 3: Seeding quality - runtime tradeoff plots
• Figure 4: Data structure for sampling from a vector $v \in \mathbb{R}^4$
• Figure 5: Relationship between $\varepsilon$ (from $\beta$ scaling) and $d_{\text{MLE}}$.

Theorems & Definitions (57)

• Definition 3
• Theorem 4
• Theorem 5
• Corollary 6
• Corollary 7
• Definition 8
• Lemma 9: Rejection sampling guarantee
• proof
• Definition 10
• Remark 11