REFRESH: Responsible and Efficient Feature Reselection Guided by SHAP Values

TL;DR

Empirical evaluations on three datasets, including a large-scale loan defaulting dataset show that REFRESH can help find alternate models with better model characteristics efficiently, and the problem of feature reselection is introduced, so that features can be selected efficiently even after a feature selection process has been done with respect to a primary objective.

Abstract

Feature selection is a crucial step in building machine learning models. This process is often achieved with accuracy as an objective, and can be cumbersome and computationally expensive for large-scale datasets. Several additional model performance characteristics such as fairness and robustness are of importance for model development. As regulations are driving the need for more trustworthy models, deployed models need to be corrected for model characteristics associated with responsible artificial intelligence. When feature selection is done with respect to one model performance characteristic (eg. accuracy), feature selection with secondary model performance characteristics (eg. fairness and robustness) as objectives would require going through the computationally expensive selection process from scratch. In this paper, we introduce the problem of feature \emph{reselection}, so that features can be selected with respect to secondary model performance characteristics efficiently even after a feature selection process has been done with respect to a primary objective. To address this problem, we propose REFRESH, a method to reselect features so that additional constraints that are desirable towards model performance can be achieved without having to train several new models. REFRESH's underlying algorithm is a novel technique using SHAP values and correlation analysis that can approximate for the predictions of a model without having to train these models. Empirical evaluations on three datasets, including a large-scale loan defaulting dataset show that REFRESH can help find alternate models with better model characteristics efficiently. We also discuss the need for reselection and REFRESH based on regulation desiderata.

Figures (7)

• Figure 1: Standard model training and the framework for REFRESH. The top block shows the conventional steps to train a model (additional steps may also be used for model training, but we show the ones most relevant to the problem). This paper introduces the feature reselection process in step 4. The bottom block describes REFRESH
• Figure 2: Ranking of groups based on secondary performance characteristic and the reselection process using this ranking.
• Figure 3: Alternate models found using REFRESH for two secondary performance characteristics: fairness ((a) and (c)) and robustness ((b) and (d)). Each point in the figure corresponds to a model trained using a different set of features. The intersection of the red lines is the baseline model. The reported metrics are the true measures and not anticipated values. The color of each point shows the number of features used to train the model. (b) and (d) show a subset of models from the fairness and robustness graphs (a) and (c) respectively, where each model has the same number of features as the baseline model.
• Figure 4: Understanding the correlation grouping based SHAP approximation. Both graphs show the anticipated model AUC's against the actual model AUC. In (a), each point on the graph represents the anticipated versus actual AUC of a model trained with all features except all features from one group. In (b), each point on the graph represents the anticipated versus actual AUC of a model trained with all features except one feature (chosen at random) from one group. The red lines show the ideal plot (where anticipated AUC $=$ actual AUC).
• Figure 5: Analysing the proposed SHAP approximation (for the home credit default risk dataset) via plotting the difference (lower is better) in actual and anticipated AUC's versus the correlation threshold chosen to form groups.