Challenges in Variable Importance Ranking Under Correlation

TL;DR

The paper tackles the problem of reliably ranking variable importance in the presence of correlated predictors. It compares standard permutation-based importance with the knockoff-based conditional predictive impact (CPI) and validates performance via a comprehensive simulation study, complemented by a theoretical result showing a fundamental limitation of the knockoff construction when correlation exceeds a critical threshold. The key theoretical finding is that in the bivariate case, the knockoff–original correlation $\\mathrm{corr}(X_1, \\tilde{X}_1)$ remains zero only for $\\mathrm{corr}(X_1, X_2) \\le 0.5$, but becomes $2\\,\\mathrm{corr}(X_1, X_2) - 1$ thereafter, which undermines the CPI's ability to adjust for correlation. The study thus demonstrates that there is no free lunch for correlated features and emphasizes careful method selection and potential alternatives such as clustering or aggregating correlated features.

Abstract

Variable importance plays a pivotal role in interpretable machine learning as it helps measure the impact of factors on the output of the prediction model. Model agnostic methods based on the generation of "null" features via permutation (or related approaches) can be applied. Such analysis is often utilized in pharmaceutical applications due to its ability to interpret black-box models, including tree-based ensembles. A major challenge and significant confounder in variable importance estimation however is the presence of between-feature correlation. Recently, several adjustments to marginal permutation utilizing feature knockoffs were proposed to address this issue, such as the variable importance measure known as conditional predictive impact (CPI). Assessment and evaluation of such approaches is the focus of our work. We first present a comprehensive simulation study investigating the impact of feature correlation on the assessment of variable importance. We then theoretically prove the limitation that highly correlated features pose for the CPI through the knockoff construction. While we expect that there is always no correlation between knockoff variables and its corresponding predictor variables, we prove that the correlation increases linearly beyond a certain correlation threshold between the predictor variables. Our findings emphasize the absence of free lunch when dealing with high feature correlation, as well as the necessity of understanding the utility and limitations behind methods in variable importance estimation.

Figures (4)

• Figure 1: Variable Importance Rankings (Scenario 1 and Scenario 2)
• Figure 2: Variable Importance Rankings (Scenario 3 and Scenario 4)
• Figure 3: CPI Estimate vs corr($X_1$, $X_2$)
• Figure 4: Corr($X_1$, $\tilde{X}_1$) vs Corr($X_1$, $X_2$)

Theorems & Definitions (2)

• Theorem 1
• proof