Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Calculation of Feature Contributions in Boosting Trees

Zhongli Jiang, Min Zhang, Dabao Zhang

TL;DR

The paper tackles global interpretability of tree ensembles by decomposing the explained variance via Shapley values of $R^2$ and introduces Q-SHAP, a polynomial-time algorithm for Shapley values of quadratic losses. It develops a single-tree core and then extends to boosting ensembles, using leaf-based reweighting and polynomial identities to compute feature contributions efficiently. Across simulations and real data, Q-SHAP achieves accurate, stable feature-specific $R^2$ estimates and dramatically outperforms SAGE and SPVIM in runtime. This work provides a scalable framework for global feature attribution in high-dimensional tree models and can generalize to broader quadratic loss functions.

Abstract

Recently, several fast algorithms have been proposed to decompose predicted value into Shapley values, enabling individualized feature contribution analysis in tree models. While such local decomposition offers valuable insights, it underscores the need for a global evaluation of feature contributions. Although coefficients of determination ($R^2$) allow for comparative assessment of individual features, individualizing $R^2$ is challenged by the underlying quadratic losses. To address this, we propose Q-SHAP, an efficient algorithm that reduces the computational complexity of calculating Shapley values for quadratic losses to polynomial time. Our simulations show that Q-SHAP not only improves computational efficiency but also enhances the accuracy of feature-specific $R^2$ estimates.

Fast Calculation of Feature Contributions in Boosting Trees

TL;DR

The paper tackles global interpretability of tree ensembles by decomposing the explained variance via Shapley values of and introduces Q-SHAP, a polynomial-time algorithm for Shapley values of quadratic losses. It develops a single-tree core and then extends to boosting ensembles, using leaf-based reweighting and polynomial identities to compute feature contributions efficiently. Across simulations and real data, Q-SHAP achieves accurate, stable feature-specific estimates and dramatically outperforms SAGE and SPVIM in runtime. This work provides a scalable framework for global feature attribution in high-dimensional tree models and can generalize to broader quadratic loss functions.

Abstract

Recently, several fast algorithms have been proposed to decompose predicted value into Shapley values, enabling individualized feature contribution analysis in tree models. While such local decomposition offers valuable insights, it underscores the need for a global evaluation of feature contributions. Although coefficients of determination () allow for comparative assessment of individual features, individualizing is challenged by the underlying quadratic losses. To address this, we propose Q-SHAP, an efficient algorithm that reduces the computational complexity of calculating Shapley values for quadratic losses to polynomial time. Our simulations show that Q-SHAP not only improves computational efficiency but also enhances the accuracy of feature-specific estimates.
Paper Structure (20 sections, 4 theorems, 34 equations, 12 figures, 6 tables, 2 algorithms)

This paper contains 20 sections, 4 theorems, 34 equations, 12 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Shapley Values of R-squared for Individual Features
  3. Model specification
  4. Empirical Estimation
  5. From R-squared to a Quadratic Loss
  6. From Quadratic Loss to Q-SHAP
  7. The Algorithm Q-SHAP for Single Trees
  8. Notations
  9. Distributing the prediction to leaves
  10. The Algorithm
  11. The Algorithm Q-SHAP for Tree Ensembles from Boosting
  12. Simulation Study
  13. Real Data Analysis
  14. Conclusion
  15. Proofs
  16. ...and 5 more sections

Key Result

Proposition 1

For any well-defined $p, n, |F|$,

Figures (12)

  • Figure 1: Illustration of (a) a decision tree built with both features $X_1$ and $X_2$ and (b) its hypothetical tree with only feature $X_1$.
  • Figure 2: The mean absolute error (MAE) averaged over 1,000 datasets with $p=100$
  • Figure 3: The running time (seconds) in simulation study
  • Figure 4: Boxplots of (a) $X_1$-specific, (b) $X_2$-specific, (c) $X_3$-specific, and (d) the sum of all feature-specific $R^2$ in the three models with $n=500$, $p=100$. The dashed lines show the theoretical $R^2$.
  • Figure 5: Boxplots of (a) $X_1$-specific, (b) $X_2$-specific, (c) $X_3$-specific, and (d) the sum of all feature-specific $R^2$ in the three models with $n=1000$, $p=100$. The dashed lines show the theoretical $R^2$.
  • ...and 7 more figures

Theorems & Definitions (4)

  • Proposition 1
  • Theorem 1
  • Proposition 2
  • Lemma 1