On marginal feature attributions of tree-based models

TL;DR

This work investigates interpretability for tree ensembles by contrasting marginal Shapley attributions with TreeSHAP. It proves that marginal feature attributions for trees are simple, piecewise-constant functions on a model-determined grid, while TreeSHAP can depend on the specific internal structure of functionally equivalent models. The authors leverage the reduced feature usage in many trees, particularly in CatBoost’s oblivious trees, to derive explicit, parameter-only formulas for marginal Shapley, Banzhaf, and Owen values, and they introduce a background-free precomputation algorithm with provable error bounds. The approach yields fast, exact marginal attributions suitable for production use, demonstrated through complexity analyses and experiments on public datasets, highlighting practical gains in efficiency and robustness for tree-ensemble explainability.

Abstract

Due to their power and ease of use, tree-based machine learning models, such as random forests and gradient-boosted tree ensembles, have become very popular. To interpret them, local feature attributions based on marginal expectations, e.g. marginal (interventional) Shapley, Owen or Banzhaf values, may be employed. Such methods are true to the model and implementation invariant, i.e. dependent only on the input-output function of the model. We contrast this with the popular TreeSHAP algorithm by presenting two (statistically similar) decision trees that compute the exact same function for which the "path-dependent" TreeSHAP yields different rankings of features, whereas the marginal Shapley values coincide. Furthermore, we discuss how the internal structure of tree-based models may be leveraged to help with computing their marginal feature attributions according to a linear game value. One important observation is that these are simple (piecewise-constant) functions with respect to a certain grid partition of the input space determined by the trained model. Another crucial observation, showcased by experiments with XGBoost, LightGBM and CatBoost libraries, is that only a portion of all features appears in a tree from the ensemble. Thus, the complexity of computing marginal Shapley (or Owen or Banzhaf) feature attributions may be reduced. This remains valid for a broader class of game values which we shall axiomatically characterize. A prime example is the case of CatBoost models where the trees are oblivious (symmetric) and the number of features in each of them is no larger than the depth. We exploit the symmetry to derive an explicit formula, with improved complexity and only in terms of the internal model parameters, for marginal Shapley (and Banzhaf and Owen) values of CatBoost models. This results in a fast, accurate algorithm for estimating these feature attributions.

Figures (10)

• Figure 1: The picture for Example \ref{['main example']} demonstrating that TreeSHAP (2018arXiv180203888L) can depend on the model make-up. Here, the features $X_1$ and $X_2$ are supported in the rectangle $\mathcal{B}=[-1,1]\times[-1,1]$ on the right which is partitioned into subrectangles $R_1=R_1^{\hbox{$-$}}\cup R_1^{\hbox{$+$}}$, $R_2$ and $R_3$. The decision trees $T_1$ and $T_2$ on the left compute the same function $g=c_1\cdot\mathbbm{1}_{R_1}+c_2\cdot\mathbbm{1}_{R_2}+c_3\cdot\mathbbm{1}_{R_3}$; the leaves are colored based on the colors of corresponding subrectangles on the right. Shapley values for various games associated with these trees are computed in Example \ref{['main example']}. In particular, over each of the subrectangles, the Shapley values arising from the marginal game, or from TreeSHAP, are constant expressions in terms of probabilities of ${\rm{P}}_\mathbf{X}(R_1^{\hbox{$-$}}),{\rm{P}}_\mathbf{X}(R_1^{\hbox{$+$}}),{\rm{P}}_\mathbf{X}(R_2), {\rm{P}}_\mathbf{X}(R_3)$ and leaf values $c_1,c_2,c_3$; see Table \ref{['Tab:main example table']}. Although the former Shapley values depend only $g$, the latter turn out to be different for $T_1$ and $T_2$. In \ref{['parameters']}, these parameters are chosen so that TreeSHAP ranks features $X_1$ and $X_2$ differently for any input from $R_2$, whereas the decision trees compute the same function and are almost indistinguishable in terms of impurity measures.
• Figure 2: The partition determined by a decision tree $T$ and its completion into a grid are demonstrated. The tree on the left implements a simple function $g=g(x_1,x_2)$ supported in $\mathcal{B}=[0,4]\times[0,3]$. In the corresponding partition $\mathscr{P}(T)$ of $\mathcal{B}$ on the right, $g$ is constant on the smaller subrectangles each corresponding to the leaf of the same color. Here, the tree is not oblivious and $\mathscr{P}(T)$ is not a grid. Adding the dotted lines refines $\mathscr{P}(T)$ to a three by four grid $\widetilde{\mathscr{P}(T)}$. On each piece of the grid, the associated marginal game (and thus its Shapley values) is almost surely constant with a value which is an expression in terms of probabilities ${\rm{P}}_{\mathbf{X}}(\tilde{R})\,(\tilde{R}\in \widetilde{\mathscr{P}(T)})$ (cf. Theorem \ref{['main theorem']}); these in general cannot be retrieved from the trained tree unless it is oblivious.
• Figure 3: The picture for Example \ref{['Repeated feature example']} which is concerned with the oblivious decision tree $T$ of depth three appearing on the left. At each split, we go right if the feature is larger than the threshold and go left otherwise. The leaves can thus be encoded with elements of $\{0,1\}^3$; the same holds for the regions into which the tree, as on the right, partitions the rectangle $\mathcal{B}=[0,3]\times[0,2]$ where the features $(X_1,X_2)$ are supported. But two of the binary codes, $001$ and $011$, do not amount to any region since the paths from the root to their corresponding leaves encounter conflicting thresholds for feature $X_1$. Any other leaf of $T$ corresponds to the region of the same color on the right.
• Figure 4: The execution times for the two steps of Algorithm \ref{['algorithm']} are depicted for CatBoost models of various depths which were trained on synthetic data \ref{['synth_data_model']} for our experiment in Section \ref{['subsec:experiments3']}. The plot on the left illustrates the training and test errors for these models. The one in the middle shows the average time it took to precompute the Shapley values for a tree from the ensemble in the logarithmic scale. Finally, the last plot captures the on-the-fly computation time for obtaining the Shapley values of 1,000 random data point based on the precomputed tables.
• Figure 5: The execution times are plotted for the interventional TreeSHAP---using the CatBoost's built-in method get_feature_importance(shap_calc_type="Regular") and background datasets $D_*$ of various sizes---and the two steps of our proposed algorithm \ref{['algorithm']}---which requires no background dataset and its accuracy is dictated by the training set $D$---once applied to CatBoost models trained for Section \ref{['subsec:experiments3']}. The plots confirm the complexity analysis outlined in Table \ref{['Tab: complexity']}.

Theorems & Definitions (70)

• Lemma 2.1
• Remark 2.2
• Example 2.3
• Theorem 2.4
• Example 2.5
• Definition 2.6
• Example 2.7
• Definition 2.8
• Example 3.1
• Theorem 3.2