Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernel Alignment for Unsupervised Feature Selection via Matrix Factorization

Ziyuan Lin, Deanna Needell

TL;DR

This work addresses unsupervised feature selection by integrating kernel alignment with matrix factorization to capture nonlinear relationships among features. It introduces KAUFS, a non-negativity constrained factorization framework with inner-product regularization to reduce feature redundancy, and extends it to MKAUFS to learn a consensus kernel via multiple kernels. The methods demonstrate strong clustering performance and reduced redundancy across diverse real-world datasets, with convergence guarantees and scalable optimization. Practically, this approach reduces the need for kernel selection and provides robust feature subsets suitable for downstream clustering and interpretation.

Abstract

By removing irrelevant and redundant features, feature selection aims to find a good representation of the original features. With the prevalence of unlabeled data, unsupervised feature selection has been proven effective in alleviating the so-called curse of dimensionality. Most existing matrix factorization-based unsupervised feature selection methods are built upon subspace learning, but they have limitations in capturing nonlinear structural information among features. It is well-known that kernel techniques can capture nonlinear structural information. In this paper, we construct a model by integrating kernel functions and kernel alignment, which can be equivalently characterized as a matrix factorization problem. However, such an extension raises another issue: the algorithm performance heavily depends on the choice of kernel, which is often unknown a priori. Therefore, we further propose a multiple kernel-based learning method. By doing so, our model can learn both linear and nonlinear similarity information and automatically generate the most appropriate kernel. Experimental analysis on real-world data demonstrates that the two proposed methods outperform other classic and state-of-the-art unsupervised feature selection methods in terms of clustering results and redundancy reduction in almost all datasets tested.

Kernel Alignment for Unsupervised Feature Selection via Matrix Factorization

TL;DR

This work addresses unsupervised feature selection by integrating kernel alignment with matrix factorization to capture nonlinear relationships among features. It introduces KAUFS, a non-negativity constrained factorization framework with inner-product regularization to reduce feature redundancy, and extends it to MKAUFS to learn a consensus kernel via multiple kernels. The methods demonstrate strong clustering performance and reduced redundancy across diverse real-world datasets, with convergence guarantees and scalable optimization. Practically, this approach reduces the need for kernel selection and provides robust feature subsets suitable for downstream clustering and interpretation.

Abstract

By removing irrelevant and redundant features, feature selection aims to find a good representation of the original features. With the prevalence of unlabeled data, unsupervised feature selection has been proven effective in alleviating the so-called curse of dimensionality. Most existing matrix factorization-based unsupervised feature selection methods are built upon subspace learning, but they have limitations in capturing nonlinear structural information among features. It is well-known that kernel techniques can capture nonlinear structural information. In this paper, we construct a model by integrating kernel functions and kernel alignment, which can be equivalently characterized as a matrix factorization problem. However, such an extension raises another issue: the algorithm performance heavily depends on the choice of kernel, which is often unknown a priori. Therefore, we further propose a multiple kernel-based learning method. By doing so, our model can learn both linear and nonlinear similarity information and automatically generate the most appropriate kernel. Experimental analysis on real-world data demonstrates that the two proposed methods outperform other classic and state-of-the-art unsupervised feature selection methods in terms of clustering results and redundancy reduction in almost all datasets tested.
Paper Structure (24 sections, 1 theorem, 46 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 24 sections, 1 theorem, 46 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Other Related Work
  3. Contribution
  4. Organization
  5. Notation
  6. The Framework of KAUFS
  7. Kernel Alignment
  8. KAUFS Model
  9. Algorithm and Convergence Analysis
  10. Algorithm
  11. Convergence Analysis
  12. Mutiple Kernel Method for KAUFS
  13. MKAUFS Model
  14. Algorithm
  15. Computational Complexity
  16. ...and 9 more sections

Key Result

Theorem 1

For $\mathbf{W},\mathbf{H} \ge 0$, the values of the objective function given in 234 are non-increasing by employing the updating rules 315 and 316.

Figures (2)

  • Figure 1: Clustering performance ACC and NMI of KAUFS and MKAUFS w.r.t $\alpha$ and $\beta$ on the WarpAR dataset.
  • Figure 2: Clustering performance ACC and NMI of KAUFS and MKAUFS w.r.t $\alpha$ and $\beta$ on the Yale64 dataset.

Theorems & Definitions (6)

  • Definition 1: Centered Kernel Function
  • Definition 2: Centered Kernel Matrix
  • Definition 3: Centered Kernel Alignment
  • Definition 4: Unnormalized Centered Kernel Alignment
  • Theorem 1
  • proof