Enhancing Kernel Power K-means: Scalable and Robust Clustering with Random Fourier Features and Possibilistic Method

TL;DR

The first approximation theory for applying random Fourier features (RFF) to KPKM is introduced, introducing the first approximation theory for applying random Fourier features (RFF) to KPKM and establishing strong theoretical guarantees for this combination.

Abstract

Kernel power $k$-means (KPKM) leverages a family of means to mitigate local minima issues in kernel $k$-means. However, KPKM faces two key limitations: (1) the computational burden of the full kernel matrix restricts its use on extensive data, and (2) the lack of authentic centroid-sample assignment learning reduces its noise robustness. To overcome these challenges, we propose RFF-KPKM, introducing the first approximation theory for applying random Fourier features (RFF) to KPKM. RFF-KPKM employs RFF to generate efficient, low-dimensional feature maps, bypassing the need for the whole kernel matrix. Crucially, we are the first to establish strong theoretical guarantees for this combination: (1) an excess risk bound of $\mathcal{O}(\sqrt{k^3/n})$, (2) strong consistency with membership values, and (3) a $(1+\varepsilon)$ relative error bound achievable using the RFF of dimension $\mathrm{poly}(\varepsilon^{-1}\log k)$. Furthermore, to improve robustness and the ability to learn multiple kernels, we propose IP-RFF-MKPKM, an improved possibilistic RFF-based multiple kernel power $k$-means. IP-RFF-MKPKM ensures the scalability of MKPKM via RFF and refines cluster assignments by combining the merits of the possibilistic membership and fuzzy membership. Experiments on large-scale datasets demonstrate the superior efficiency and clustering accuracy of the proposed methods compared to the state-of-the-art alternatives.

Figures (5)

• Figure 1: Convergence analysis and sensitivity analysis of $s_0$ and $\lambda$ of IP-RFF-MKPKM on Caltech101-7 dataset. Convergence and sensitivity studies on other benchmark datasets are given in the Appendix.
• Figure 2: Impact of RFF dimension on RFF-KPKM clustering accuracy on six benchmark datasets. The RFF dimension varies from 5 to 100.
• Figure 3: Logarithmic running time comparison of IP-RFF-MKPKM with eight benchmark methods on seven benchmark datasets. Bar absence denotes an out-of-memory runtime exception during method execution.
• Figure 4: Convergence analysis of IP-RFF-MKPKM on six benchmark datasets.
• Figure 5: Sensitivity analysis of $s_0$ and $\lambda$ of IP-RFF-MKPKM on six benchmark datasets.

Theorems & Definitions (15)

• Theorem 1
• Theorem 2
• Theorem 3
• Lemma 1
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4
• proof