Achieving interpretable machine learning by functional decomposition of black-box models into explainable predictor effects

TL;DR

This paper tackles the interpretability gap of high-performing black-box predictors by proposing a functional decomposition of the prediction function into interpretable subfunctions, implemented via a neural additive model (NAM) surrogate. The key novelty is stacked orthogonality, a level-wise purity constraint that yields a decomposition $F(X) = μ + ∑_{|θ|=1} f_θ(X_θ) + ∑_{|θ|=2} f_θ(X_θ) + …$ with interpretable main effects and interactions, and level-wise explainable variance $I_k$. A three-step NAM-based estimation pipeline plus post-hoc orthogonalization enforces stacked orthogonality and enables stable, ensemble-averaged estimates of the subfunctions. Synthetic experiments demonstrate accurate recovery of the true subfunctions and quantify interpretability gains, suggesting practical utility for transparent ML in domains demanding explanations of feature effects and interactions.

Abstract

Machine learning (ML) has seen significant growth in both popularity and importance. The high prediction accuracy of ML models is often achieved through complex black-box architectures that are difficult to interpret. This interpretability problem has been hindering the use of ML in fields like medicine, ecology and insurance, where an understanding of the inner workings of the model is paramount to ensure user acceptance and fairness. The need for interpretable ML models has boosted research in the field of interpretable machine learning (IML). Here we propose a novel approach for the functional decomposition of black-box predictions, which is considered a core concept of IML. The idea of our method is to replace the prediction function by a surrogate model consisting of simpler subfunctions. Similar to additive regression models, these functions provide insights into the direction and strength of the main feature contributions and their interactions. Our method is based on a novel concept termed stacked orthogonality, which ensures that the main effects capture as much functional behavior as possible and do not contain information explained by higher-order interactions. Unlike earlier functional IML approaches, it is neither affected by extrapolation nor by hidden feature interactions. To compute the subfunctions, we propose an algorithm based on neural additive modeling and an efficient post-hoc orthogonalization procedure.

Figures (5)

• Figure 1: Illustration of the neural additive model in Eq. \ref{['eq:NAM']}. In the example considered here, there are two features $X_1$ and $X_2$. Accordingly, the set of functions $f_\theta^0$, $\theta\in\mathcal{P}(\Upsilon)\backslash\emptyset$, is given by the two main effects $f_1^0(X_1)$, $f_2^0 (X_2)$ and the two-way interaction $f_{12}^0(X_1, X_2)$. Each function is represented by a fully connected artificial neural network (ANN). The units in the penultimate layers of the ANNs are denoted by $U_1 \in \mathbb{R}^{b_1}$, $U_2 \in \mathbb{R}^{b_2}$ and $U_{12} \in \mathbb{R}^{b_{12}}$, where $b_1$, $b_2$ and $b_{12}$ are the widths of the layers. The outputs of the ANNs are given by the dot products $U_1^\top \mathbf{w}_1^0$, $U_2^\top \mathbf{w}_2^0$ and $U_{12}^\top \mathbf{w}_{12}^0$, where $\mathbf{w}_1^0$, $\mathbf{w}_2^0$ and $\mathbf{w}_{12}^0$ are vectors of weights. The prediction function $F(X_1, X_2)$ is given by the sum of the three dot products (hence the term neural additive model). The parameters of the ANNs are estimated jointly by backpropagation. Details on model fitting and the specification of the ANN architectures are given in Appendix A.
• Figure 2: Experiments with synthetic data. The blue lines visualize the main effects $f_1(X_1), f_2(X_2), f_3(X_3)$, as obtained by applying the proposed three-step algorithm to samples of size $n=2000$ each. The black lines correspond to the true post-hoc-orthogonalized main effects defined in Appendix C (A1-A3: scenario 1, B1-B3: scenario 2, C1-C3: scenario 3).
• Figure 3: Experiments with synthetic data. The blue lines visualize the main effects $f_1(X_1), f_2(X_2), f_3(X_3)$, as obtained by applying the proposed three-step algorithm to samples of size $n=5000$. The black lines correspond to the true post-hoc-orthogonalized main effects defined in Appendix C (A1-A3: scenario 1, B1-B3: scenario 2, C1-C3: scenario 3).
• Figure 4: Experiments with synthetic data. The left column presents the average two-way interaction effects $f_{12}(X_{12}), f_{13}(X_{13}), f_{23}(X_{23})$, as obtained by applying the proposed three-step algorithm to samples of size $n=2000$. The right columns contain the true post-hoc-orthogonalized two-way interactions defined in Appendix C.
• Figure 5: Experiments with synthetic data. The left column presents the average two-way interaction effects $f_{12}(X_{12}), f_{13}(X_{13}), f_{23}(X_{23})$, as obtained by applying the proposed three-step algorithm to samples of size $n=5000$. The right columns contain the true post-hoc-orthogonalized two-way interactions defined in Appendix C.