Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Binary Feature Mask Optimization for Feature Selection

Mehmet E. Lorasdagi, Mehmet Y. Turali, Suleyman S. Kozat

TL;DR

This paper addresses the challenge of feature selection for generic machine learning models without re-training the model during the selection process. It introduces a binary feature masking framework and two algorithms, GBMO and FLBMO, which optimize feature subsets by evaluating model predictions on validation data and unseen data, using a Hadamard-mask operation to select features. The approach yields robust improvements over traditional methods on both synthetic and real datasets, with GBMO offering adaptive feature counts and FLBMO excelling when a fixed feature count is required; both are demonstrated with LightGBM and MLP and are published as open-source. Overall, the work provides a practical, training-efficient solution for holistic feature-subset selection that accounts for interactions among features and model-specific performance, enabling better generalization with fewer features.

Abstract

We investigate feature selection problem for generic machine learning models. We introduce a novel framework that selects features considering the outcomes of the model. Our framework introduces a novel feature masking approach to eliminate the features during the selection process, instead of completely removing them from the dataset. This allows us to use the same machine learning model during feature selection, unlike other feature selection methods where we need to train the machine learning model again as the dataset has different dimensions on each iteration. We obtain the mask operator using the predictions of the machine learning model, which offers a comprehensive view on the subsets of the features essential for the predictive performance of the model. A variety of approaches exist in the feature selection literature. However, to our knowledge, no study has introduced a training-free framework for a generic machine learning model to select features while considering the importance of the feature subsets as a whole, instead of focusing on the individual features. We demonstrate significant performance improvements on the real-life datasets under different settings using LightGBM and Multi-Layer Perceptron as our machine learning models. The high performance of our General Binary Mask Optimization algorithm stems from its feature masking approach to select features and its flexibility in the number of selected features. The algorithm selects features based on the validation performance of the machine learning model. Hence, the number of selected features is not predetermined and adjusts dynamically to the dataset. Additionally, we openly share the implementation or our code to encourage further research in this area.

Binary Feature Mask Optimization for Feature Selection

TL;DR

This paper addresses the challenge of feature selection for generic machine learning models without re-training the model during the selection process. It introduces a binary feature masking framework and two algorithms, GBMO and FLBMO, which optimize feature subsets by evaluating model predictions on validation data and unseen data, using a Hadamard-mask operation to select features. The approach yields robust improvements over traditional methods on both synthetic and real datasets, with GBMO offering adaptive feature counts and FLBMO excelling when a fixed feature count is required; both are demonstrated with LightGBM and MLP and are published as open-source. Overall, the work provides a practical, training-efficient solution for holistic feature-subset selection that accounts for interactions among features and model-specific performance, enabling better generalization with fewer features.

Abstract

We investigate feature selection problem for generic machine learning models. We introduce a novel framework that selects features considering the outcomes of the model. Our framework introduces a novel feature masking approach to eliminate the features during the selection process, instead of completely removing them from the dataset. This allows us to use the same machine learning model during feature selection, unlike other feature selection methods where we need to train the machine learning model again as the dataset has different dimensions on each iteration. We obtain the mask operator using the predictions of the machine learning model, which offers a comprehensive view on the subsets of the features essential for the predictive performance of the model. A variety of approaches exist in the feature selection literature. However, to our knowledge, no study has introduced a training-free framework for a generic machine learning model to select features while considering the importance of the feature subsets as a whole, instead of focusing on the individual features. We demonstrate significant performance improvements on the real-life datasets under different settings using LightGBM and Multi-Layer Perceptron as our machine learning models. The high performance of our General Binary Mask Optimization algorithm stems from its feature masking approach to select features and its flexibility in the number of selected features. The algorithm selects features based on the validation performance of the machine learning model. Hence, the number of selected features is not predetermined and adjusts dynamically to the dataset. Additionally, we openly share the implementation or our code to encourage further research in this area.
Paper Structure (16 sections, 19 equations, 4 figures, 8 tables, 3 algorithms)

This paper contains 16 sections, 19 equations, 4 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Description
  3. Feature Masking Framework for Feature Selection
  4. General Binary Mask Optimization (GBMO)
  5. Binary Mask Optimization For a Determined Number of Features (FLBMO)
  6. Experiments
  7. Datasets
  8. Connectionist Bench
  9. Residential Building
  10. Models and Hyperparameter Tuning
  11. Synthetic Data Experiments
  12. Real Life Data Experiments
  13. Discussion
  14. Methodology Limitations
  15. Conclusion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: The plot of the number of the remaining features for the GBMO algorithm with respect to the iteration numbers on the synthetic dataset with LightGBM model.
  • Figure 2: The plot of the validation losses for the GBMO algorithm with respect to the iteration numbers on the synthetic dataset with LightGBM model.
  • Figure 3: The plot of the number of remaining features for the GBMO algorithm with respect to the iteration numbers on the Residential Building dataset with MLP model.
  • Figure 4: The plot of the validation losses for the GBMO algorithm with respect to the iteration numbers on the Residential Building dataset with MLP model.