Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An NEPv Approach for Feature Selection via Orthogonal OCCA with the (2,1)-norm Regularization

Li Wang, Lei-Hong Zhang, Ren-Cang Li

TL;DR

This work addresses supervised feature selection in high-dimensional settings by introducing OCCA21, an orthogonal canonical correlation analysis model regularized with the $\|P\|_{2,1}$ norm to induce row sparsity. The authors formulate the optimization on the Stiefel manifold and solve it via a nonlinear eigenvalue problem (NEPv) framework, yielding the OCCA-FS feature selector. They show an exact connection to the OLSR-OS model with an optimally eliminated scaling parameter, and develop a practical NEPv approach with a perturbed objective $f_{\varepsilon_0}$ and a LOCG-accelerated solver to ensure monotone convergence and scalability. Extensive experiments across six real datasets demonstrate superior classification performance and robust, globally convergent optimization compared to existing methods such as PEB-FS, while highlighting and correcting optimization issues in prior alternating schemes. The proposed method provides a scalable, theoretically grounded pipeline for high-dimensional feature ranking and selection with tangible improvements in predictive accuracy.

Abstract

A novel feature selection model via orthogonal canonical correlation analysis with the $(2,1)$-norm regularization is proposed, and the model is solved by a practical NEPv approach (nonlinear eigenvalue problem with eigenvector dependency), yielding a feature selection method named OCCA-FS. It is proved that OCCA-FS always produces a sequence of approximations with monotonic objective values and is globally convergent. Extensive numerical experiments are performed to compare OCCA-FS against existing feature selection methods. The numerical results demonstrate that OCCA-FS produces superior classification performance and often comes out on the top among all feature selection methods in comparison.

An NEPv Approach for Feature Selection via Orthogonal OCCA with the (2,1)-norm Regularization

TL;DR

This work addresses supervised feature selection in high-dimensional settings by introducing OCCA21, an orthogonal canonical correlation analysis model regularized with the norm to induce row sparsity. The authors formulate the optimization on the Stiefel manifold and solve it via a nonlinear eigenvalue problem (NEPv) framework, yielding the OCCA-FS feature selector. They show an exact connection to the OLSR-OS model with an optimally eliminated scaling parameter, and develop a practical NEPv approach with a perturbed objective and a LOCG-accelerated solver to ensure monotone convergence and scalability. Extensive experiments across six real datasets demonstrate superior classification performance and robust, globally convergent optimization compared to existing methods such as PEB-FS, while highlighting and correcting optimization issues in prior alternating schemes. The proposed method provides a scalable, theoretically grounded pipeline for high-dimensional feature ranking and selection with tangible improvements in predictive accuracy.

Abstract

A novel feature selection model via orthogonal canonical correlation analysis with the -norm regularization is proposed, and the model is solved by a practical NEPv approach (nonlinear eigenvalue problem with eigenvector dependency), yielding a feature selection method named OCCA-FS. It is proved that OCCA-FS always produces a sequence of approximations with monotonic objective values and is globally convergent. Extensive numerical experiments are performed to compare OCCA-FS against existing feature selection methods. The numerical results demonstrate that OCCA-FS produces superior classification performance and often comes out on the top among all feature selection methods in comparison.

Paper Structure

This paper contains 15 sections, 5 theorems, 56 equations, 4 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. OCCA vs. OLSR
  3. Connection to OLSR-OS
  4. Solving regularized OLSR-OS
  5. NEPv for OCCA with the (2,1)-norm regularization
  6. The NEPv Approach
  7. A practical NEPv approach
  8. Acceleration via LOCG
  9. Numerical Experiments
  10. Experimental setting
  11. Classification performance
  12. Empirical illustration on convergence by PEB-FS and OCCA-FS
  13. Parameter sensitivity analysis
  14. Comparisons of plain NEPv with accelerated NEPv
  15. Conclusion

Key Result

Theorem 3.1

Let $P\in\mathbb{O}^{n\times k}$ and rearrange $\{\|P_{(i,:)}\|_2\}_{i=1}^n$ descendingly as Then

Figures (4)

  • Figure 4.1: Behavior of objective values during optimization processes in OCCA-FS and PEB-FS. In terms of optimization, eventually, the smaller the objective value is, the more superior the optimization solver will be. For dataset Yale, PEB-FS does not produce monotonically decreasing objective values as claimed by Zhang2018.
  • Figure 4.2: Parameter sensitivity analysis on OCCA-FS with respect to testing accuracy via varying the number $q$ of selected features and regularization parameter $\alpha$ on the six datasets.
  • Figure 4.3: Comparison between NEPv and AccNEPv in terms of objective value during the NEPv optimization processes and classification accuracy on datasets COIL20-$t$ for $t\in\{1, 2, 3, 4, 5, 6\}$.
  • Figure 4.4: Experimental results by AccNEPv on the four text datasets in \ref{['tab:text-data']} in terms of both objective value during optimization iterations and classification accuracy.

Theorems & Definitions (9)

  • Theorem 3.1
  • proof
  • Theorem 3.2: li:2024
  • Theorem 3.3: li:2024
  • Theorem 3.4: li:2024
  • Theorem 3.5
  • proof
  • Remark 3.1
  • Remark 3.2