Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLpowershap: Logistic Loss-based Automated Shapley Values Feature Selection Method

Iqbal Madakkatel, Elina Hyppönen

TL;DR

LLpowershap is a viable wrapper feature selection method that can be used for feature selection in large biomedical datasets and other settings and is ranked the best in mean ranking among the seven feature selection methods tested on the benchmark datasets.

Abstract

Shapley values have been used extensively in machine learning, not only to explain black box machine learning models, but among other tasks, also to conduct model debugging, sensitivity and fairness analyses and to select important features for robust modelling and for further follow-up analyses. Shapley values satisfy certain axioms that promote fairness in distributing contributions of features toward prediction or reducing error, after accounting for non-linear relationships and interactions when complex machine learning models are employed. Recently, a number of feature selection methods utilising Shapley values have been introduced. Here, we present a novel feature selection method, LLpowershap, which makes use of loss-based Shapley values to identify informative features with minimal noise among the selected sets of features. Our simulation results show that LLpowershap not only identifies higher number of informative features but outputs fewer noise features compared to other state-of-the-art feature selection methods. Benchmarking results on four real-world datasets demonstrate higher or at par predictive performance of LLpowershap compared to other Shapley based wrapper methods, or filter methods.

LLpowershap: Logistic Loss-based Automated Shapley Values Feature Selection Method

TL;DR

LLpowershap is a viable wrapper feature selection method that can be used for feature selection in large biomedical datasets and other settings and is ranked the best in mean ranking among the seven feature selection methods tested on the benchmark datasets.

Abstract

Shapley values have been used extensively in machine learning, not only to explain black box machine learning models, but among other tasks, also to conduct model debugging, sensitivity and fairness analyses and to select important features for robust modelling and for further follow-up analyses. Shapley values satisfy certain axioms that promote fairness in distributing contributions of features toward prediction or reducing error, after accounting for non-linear relationships and interactions when complex machine learning models are employed. Recently, a number of feature selection methods utilising Shapley values have been introduced. Here, we present a novel feature selection method, LLpowershap, which makes use of loss-based Shapley values to identify informative features with minimal noise among the selected sets of features. Our simulation results show that LLpowershap not only identifies higher number of informative features but outputs fewer noise features compared to other state-of-the-art feature selection methods. Benchmarking results on four real-world datasets demonstrate higher or at par predictive performance of LLpowershap compared to other Shapley based wrapper methods, or filter methods.
Paper Structure (17 sections, 5 equations, 3 figures, 4 tables, 3 algorithms)

This paper contains 17 sections, 5 equations, 3 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Other feature selection methods
  4. LLpowershap
  5. Automatic Mode
  6. Experiments
  7. Feature Selection Methods
  8. Simulation Data
  9. Benchmark Datasets
  10. Results
  11. Simulation Data
  12. Benchmark Datasets
  13. Discussion
  14. Conclusion
  15. Acknowledgements.
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Simulation benchmark results on the datasets created using the make_classification of sklearn for 5,000 samples with five different random seeds. X-axis labels show the percentages of informative features in the datasets with the counts in brackets. Each dot of a particular colour represents result from one simulation (for example, using random seed 0) for a particular method.
  • Figure 2: Benchmark performance using default CatBoost model, with error bars representing the standard deviations.
  • Figure 3: Benchmark performance using default LightGBM model, with error bars representing the standard deviations.