Consistent Individualized Feature Attribution for Tree Ensembles

TL;DR

The paper tackles the problem of inconsistent feature attribution in tree ensembles and presents SHAP values as the unique, locally accurate additive attribution framework. It introduces Tree SHAP to compute exact SHAP values in polynomial time and extends the approach to SHAP interaction values, enabling robust main and interaction effects. Through user studies, performance benchmarks, and novel applications (supervised clustering, SHAP plots), the work demonstrates improved interpretability and scalability, with practical integration into XGBoost and LightGBM. This advances reliable, human-aligned explanations for complex tree-based models and provides tools that outperform traditional attribution methods in both accuracy and usability.

Abstract

Interpreting predictions from tree ensemble methods such as gradient boosting machines and random forests is important, yet feature attribution for trees is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are inconsistent, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This is a fundamental problem that casts doubt on any comparison between features. To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (SHapley Additive exPlanation) values, which are the unique consistent and locally accurate attribution values. We then extend SHAP values to interaction effects and define SHAP interaction values. We propose a rich visualization of individualized feature attributions that improves over classic attribution summaries and partial dependence plots, and a unique "supervised" clustering (clustering based on feature attributions). We demonstrate better agreement with human intuition through a user study, exponential improvements in run time, improved clustering performance, and better identification of influential features. An implementation of our algorithm has also been merged into XGBoost and LightGBM, see http://github.com/slundberg/shap for details.

Figures (10)

• Figure 1: Two simple tree models that demonstrate inconsistencies in the Saabas, gain, and split count attribution methods: The Cough feature has a larger impact in Model B than Model A, but is attributed less importance in Model B. Similarly, the Cough feature has a larger impact than Fever in Model B, yet is attributed less importance. The individualized attributions explain a single prediction of the model (when both Cough and Fever are Yes) by allocating the difference between the expected value of the model's output (20 for Model A, 25 for Model B) and the current output (80 for Model A, 90 for Model B). The global attributions represent the overall importance of a feature in the model. Without consistency it is impossible to reliably compare feature attribution values.
• Figure 2: SHAP (SHapley Additive exPlanation) values explain the output of a function $f$ as a sum of the effects $\phi_i$ of each feature being introduced into a conditional expectation. Importantly, for non-linear functions the order in which features are introduced matters. SHAP values result from averaging over all possible orderings. Proofs from game theory show this is the only possible consistent approach where $\sum_{i=0}^M \phi_i = f(x)$. In contrast, the only current individualized feature attribution method for trees satisfies the summation, but is inconsistent because it only considers a single ordering path_blog.
• Figure 3: Feature attribution values from 34 participants shown the tree from Model A in Figure \ref{['fig:and_trees']}. The first number represents the allocation to the Fever feature, while the second represents the allocation to the Cough feature. Participants from Amazon Mechanical Turk were not selected for machine learning expertise. No constraints were placed on the feature attribution values users entered.
• Figure 4: Supervised clustering with SHAP feature attributions in the UCI census dataset identifies among 2,000 individuals distinct subgroups of people that share similar reasons for making money. An XGBoost model with 500 trees of max depth six was trained on demographic data using a shrinkage factor of $\eta = 0.005$. This model was then used to predict the log odds that each person makes $\ge\$50K$. Each prediction was explained using Tree SHAP, and then clustered using hierarchical agglomerative clustering (imagine a dendrogram above the plot joining the samples). Red feature attributions push the score higher, while blue feature attributions push the score lower (as in Figure \ref{['fig:number_line']} but rotated 90$^\circ$). A few of the noticeable subgroups are annotated with the features that define them.
• Figure 5: Runtime improvement of Algorithm 2 over using Equation \ref{['eq:shapley']} and Algorithm 1. An XGBoost model with 50 trees was trained using an equally increasing number of input features and max tree depths. The time to explain one input vector is reported.

Theorems & Definitions (1)

• definition 1