Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lazy FSCA for Unsupervised Variable Selection

Federico Zocco, Marco Maggipinto, Gian Antonio Susto, Seán McLoone

TL;DR

This paper tackles unsupervised variable selection by proposing a lazy greedy variant of Forward Selection Component Analysis (L-FSCA) to efficiently approximate FSCA. It situates VE-based selection within a submodular-optimization framework, clarifying when theoretical guarantees apply and how lazy greedy accelerates computation. Through extensive experiments on simulated and real datasets, L-FSCA delivers nearly identical performance to FSCA while achieving substantial speed-ups (roughly 22%–94%), and positions VE-/FP-/MI-based methods in a spectrum of strengths across variance explained, information gain, and orthogonality. The results support using L-FSCA as a practical, scalable choice for unsupervised variable selection with strong data compression capabilities, and the work provides a comprehensive benchmark and public code for future comparisons.

Abstract

Various unsupervised greedy selection methods have been proposed as computationally tractable approximations to the NP-hard subset selection problem. These methods rely on sequentially selecting the variables that best improve performance with respect to a selection criterion. Theoretical results exist that provide performance bounds and enable "lazy greedy" efficient implementations for selection criteria that satisfy a diminishing returns property known as submodularity. This has motivated the development of variable selection algorithms based on mutual information and frame potential. Recently, the authors introduced Forward Selection Component Analysis (FSCA) which uses variance explained as its selection criterion. While this criterion is not submodular, FSCA has been shown to be highly effective for applications such as measurement plan optimisation. In this paper a "lazy" implementation of the FSCA algorithm (L-FSCA) is proposed, which, although not equivalent to FSCA due to the absence of submodularity, has the potential to yield comparable performance while being up to an order of magnitude faster to compute. The efficacy of L-FSCA is demonstrated by performing a systematic comparison with FSCA and five other unsupervised variable selection methods from the literature using simulated and real-world case studies. Experimental results confirm that L-FSCA yields almost identical performance to FSCA while reducing computation time by between 22% and 94% for the case studies considered.

Lazy FSCA for Unsupervised Variable Selection

TL;DR

This paper tackles unsupervised variable selection by proposing a lazy greedy variant of Forward Selection Component Analysis (L-FSCA) to efficiently approximate FSCA. It situates VE-based selection within a submodular-optimization framework, clarifying when theoretical guarantees apply and how lazy greedy accelerates computation. Through extensive experiments on simulated and real datasets, L-FSCA delivers nearly identical performance to FSCA while achieving substantial speed-ups (roughly 22%–94%), and positions VE-/FP-/MI-based methods in a spectrum of strengths across variance explained, information gain, and orthogonality. The results support using L-FSCA as a practical, scalable choice for unsupervised variable selection with strong data compression capabilities, and the work provides a comprehensive benchmark and public code for future comparisons.

Abstract

Various unsupervised greedy selection methods have been proposed as computationally tractable approximations to the NP-hard subset selection problem. These methods rely on sequentially selecting the variables that best improve performance with respect to a selection criterion. Theoretical results exist that provide performance bounds and enable "lazy greedy" efficient implementations for selection criteria that satisfy a diminishing returns property known as submodularity. This has motivated the development of variable selection algorithms based on mutual information and frame potential. Recently, the authors introduced Forward Selection Component Analysis (FSCA) which uses variance explained as its selection criterion. While this criterion is not submodular, FSCA has been shown to be highly effective for applications such as measurement plan optimisation. In this paper a "lazy" implementation of the FSCA algorithm (L-FSCA) is proposed, which, although not equivalent to FSCA due to the absence of submodularity, has the potential to yield comparable performance while being up to an order of magnitude faster to compute. The efficacy of L-FSCA is demonstrated by performing a systematic comparison with FSCA and five other unsupervised variable selection methods from the literature using simulated and real-world case studies. Experimental results confirm that L-FSCA yields almost identical performance to FSCA while reducing computation time by between 22% and 94% for the case studies considered.

Paper Structure

This paper contains 23 sections, 30 equations, 3 figures, 12 tables, 9 algorithms.

Table of Contents

  1. Introduction
  2. Underpinning Theory
  3. Theoretical Guarantees on Performance
  4. Submodularity and the Lazy Greedy Implementation
  5. Candidate Algorithms, Complexity and Performance Metrics
  6. Candidate Algorithms
  7. Forward Selection Component Analysis (FSCA)
  8. Forward Orthogonal Search Maximizing the Overall Dependency (FOS-MOD)
  9. Information Theoretic Feature Selection (ITFS)
  10. Principal Feature Selection (PFS)
  11. Forward Selection Frame Potential (FSFP)
  12. Unsupervised Forward Selection (UFS)
  13. Computational Complexity
  14. Performance Metrics
  15. Results
  16. ...and 8 more sections

Figures (3)

  • Figure 1: A taxonomy of approximate solution methods (with examples) for the NP-hard variable/feature selection problem.
  • Figure 2: Variance explained as a function of the number of selected variables, $k$, for the 'Simulated 1' case study
  • Figure 3: VE as a function of the number of selected variables $k$ for the larger datasets.