Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discovery and inference beyond linearity by integrating Bayesian regression, tree ensembles and Shapley values

Giorgio Spadaccini, Marjolein Fokkema, Mark A. van de Wiel

TL;DR

Addressing the need for uncertainty-aware, hypothesis-free discovery of nonlinear and interactive factors in healthcare, the paper targets reliable inference for local feature effects in ML. It introduces RuleSHAP, which fuses Bayesian sparse regression with a rule-based generator and Shapley attribution, and derives an efficient formula for marginal Shapley values $\phi_j(x^*)$ with posterior uncertainty. The empirical evaluation shows RuleSHAP reconstructs linear effects, detects beyond-linear interactions, and provides calibrated local inference, outperforming RuleFit, HorseRule, and RF in local inference across simulations and HELIUS data. Applied to the HELIUS cohort, RuleSHAP uncovers nonlinear interactions among age, ethnicity, sex, BMI and glucose affecting high cholesterol and systolic BP, demonstrating practical epidemiological utility.

Abstract

Machine Learning (ML) is gaining popularity for hypothesis-free discovery of risk and protective factors in healthcare studies. ML is strong at discovering nonlinearities and interactions, but this power is compromised by a lack of reliable inference. Although Shapley values provide local measures of features' effects, valid uncertainty quantification for these effects is typically lacking, thus precluding statistical inference. We propose RuleSHAP, a framework that addresses this limitation by combining a dedicated Bayesian sparse regression model with a new tree-based rule generator and Shapley value attribution. RuleSHAP provides detection of nonlinear and interaction effects with uncertainty quantification at the individual level. We derive an efficient formula for computing marginal Shapley values within this framework. We demonstrate the validity of our framework on simulated data. Finally, we apply RuleSHAP to data from an epidemiological cohort to detect and infer several effects for high cholesterol and blood pressure, such as nonlinear interaction effects between features like age, sex, ethnicity, BMI and glucose level.

Discovery and inference beyond linearity by integrating Bayesian regression, tree ensembles and Shapley values

TL;DR

Addressing the need for uncertainty-aware, hypothesis-free discovery of nonlinear and interactive factors in healthcare, the paper targets reliable inference for local feature effects in ML. It introduces RuleSHAP, which fuses Bayesian sparse regression with a rule-based generator and Shapley attribution, and derives an efficient formula for marginal Shapley values with posterior uncertainty. The empirical evaluation shows RuleSHAP reconstructs linear effects, detects beyond-linear interactions, and provides calibrated local inference, outperforming RuleFit, HorseRule, and RF in local inference across simulations and HELIUS data. Applied to the HELIUS cohort, RuleSHAP uncovers nonlinear interactions among age, ethnicity, sex, BMI and glucose affecting high cholesterol and systolic BP, demonstrating practical epidemiological utility.

Abstract

Machine Learning (ML) is gaining popularity for hypothesis-free discovery of risk and protective factors in healthcare studies. ML is strong at discovering nonlinearities and interactions, but this power is compromised by a lack of reliable inference. Although Shapley values provide local measures of features' effects, valid uncertainty quantification for these effects is typically lacking, thus precluding statistical inference. We propose RuleSHAP, a framework that addresses this limitation by combining a dedicated Bayesian sparse regression model with a new tree-based rule generator and Shapley value attribution. RuleSHAP provides detection of nonlinear and interaction effects with uncertainty quantification at the individual level. We derive an efficient formula for computing marginal Shapley values within this framework. We demonstrate the validity of our framework on simulated data. Finally, we apply RuleSHAP to data from an epidemiological cohort to detect and infer several effects for high cholesterol and blood pressure, such as nonlinear interaction effects between features like age, sex, ethnicity, BMI and glucose level.
Paper Structure (28 sections, 8 theorems, 140 equations, 23 figures, 1 table, 2 algorithms)

This paper contains 28 sections, 8 theorems, 140 equations, 23 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. RuleFit family of models
  3. RuleSHAP
  4. Linear fit
  5. Rule generation
  6. Feature effect measures
  7. Direct computation of Shapley values
  8. Empirical evaluation
  9. Evaluation of feature effect reconstruction
  10. Method
  11. Results: Estimation and prediction
  12. Evaluation of uncertainty quantification
  13. Method
  14. Results: Local inference
  15. Risk and protective factors in Cardiovascular Health
  16. ...and 13 more sections

Key Result

Theorem 1

Assume to have a dataset $\mathcal{T}$ of size $n$. Consider a 0-1 coded rule decomposed as the product of single conditions and thus of the form $r(x_1,\ldots,x_p)=\prod_{k=1}^pR_k(x_k)$, with ${R_k:\mathbb{R}\rightarrow\{0,1\}}$. Given $j \in \{1,\ldots,p\}$ and a datapoint $x^*$, the contribution where $q: \mathbb{R}^p \rightarrow \mathbb{N}$ is defined as $q(x)=\sum_{k=1}^pR_k(x_k)$ and $\vee$

Figures (23)

  • Figure 1: Example of one of the many trees generated from a tree ensemble.
  • Figure 2: Example of Shapley values estimated from BART (first row), HorseRule (second row: HR1, third row: HR2) and RuleSHAP (last row) fitted on the same $n=1'000$ Friedman-generated observations. Point-wise credible intervals are color-coded based on whether they contain zero (grey) or not (light blue).
  • Figure 3: Rescaled mean squared distance between feature effects estimated by the model and the target effects for five replicates of different models fitted on Friedman-generated data, for $p=10$ features and some of the different sample sizes $n$. Full figure is shown in the Supplementary Material as Figure \ref{['fig:LocImpDistsp10']}.
  • Figure 4: Density plot for rejection rates: the average proportion of significant Shapley values. Noise features (left hand side) and signal features (right hand side) are aggregated separately. The plot shows the variability across the 100 replicates of the experiment and compares BART, HorseRule (HR1 and HR2) and RuleSHAP.
  • Figure 5: Test MSE across five fits of OLS linear regression, LASSO regression, RuleFit regression, HorseRule regression (HR1 for default settings; HR2 for custom settings), RuleSHAP regression and Random Forest. These models are fitted on subsamples of different sizes $n$ taken from the HELIUS study data, with predicted outcome of cholesterol (top row) and systolic blood pressure (bottom row). Both outcomes have unit variance.
  • ...and 18 more figures

Theorems & Definitions (19)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Example 1
  • Remark 1
  • Example 2
  • Lemma 3
  • proof
  • Lemma 4
  • ...and 9 more