Statistical Inference for Explainable Boosting Machines

TL;DR

This work develops an uncertainty-aware framework for Explainable Boosting Machines (EBMs) by adopting Boulevard regularization, which yields a feature-wise kernel ridge regression limit and an associated central limit theorem. By proving CLTs for multiple EBM variants and introducing per-feature kernels, the authors enable principled confidence and prediction intervals whose computation scales in bin-space independent of the sample size. They show minimax-optimal $O(p n^{-2/3})$ mean-squared error for Lipschitz $p$-dimensional GAMs and provide practical interval construction methods for each univariate component and the overall prediction. The approach improves interpretability and reliability of EBMs, delivering scalable inference with competitive predictive performance on real data and potential for broader additive-model extensions and discrete outcomes.

Abstract

Explainable boosting machines (EBMs) are popular "glass-box" models that learn a set of univariate functions using boosting trees. These achieve explainability through visualizations of each feature's effect. However, unlike linear model coefficients, uncertainty quantification for the learned univariate functions requires computationally intensive bootstrapping, making it hard to know which features truly matter. We provide an alternative using recent advances in statistical inference for gradient boosting, deriving methods for statistical inference as well as end-to-end theoretical guarantees. Using a moving average instead of a sum of trees (Boulevard regularization) allows the boosting process to converge to a feature-wise kernel ridge regression. This produces asymptotically normal predictions that achieve the minimax-optimal mean squared error for fitting Lipschitz GAMs with $p$ features at rate $O(pn^{-2/3})$, successfully avoiding the curse of dimensionality. We then construct prediction intervals for the response and confidence intervals for each learned univariate function with a runtime independent of the number of datapoints, enabling further explainability within EBMs.

Figures (7)

• Figure 1: Example of the feature-specific confidence intervals generated by Algorithm \ref{['alg:brebm_ident']}, compared to the point estimates from EBM nori2019interpretmlunifiedframeworkmachine. This visualizes centered feature effects on the UCI Machine Learning Reposity Obesity estimation_of_obesity_levels_based_on_eating_habits_and_physical_condition__544 dataset, when predicting weight as the response.
• Figure 2: Example of confidence and prediction intervals on a simple 1D function $y=2\sin(2\pi x) + x^2$.
• Figure 3: Per-feature CIs and coverages for $f(x)=-5+10\sin(\pi x^{(1)}) + 5\cos(\pi x^{(2)}) + 20(x^{(2)}-0.5)^2 +10x^{(3)} - 5x^{(4)}$.
• Figure 4: MSE horse races on UCI machine learning repository datasets. All hyperparameters tuned by Optuna.
• Figure 5: RMSE for Algorithm \ref{['alg:brebm_ident']} and benchmarks with 2 s.e. bands over 50 trials, on $f(x) = \sin(4\pi x)$ and the diabetes dataset from Efron_2004. Both EBM and Algorithm \ref{['alg:brebm_ident']} are highly resistant to overfitting.

Theorems & Definitions (52)

• Definition 4.1: Regression trees
• Definition 4.2: Structure vectors and matrices
• Theorem 4.8
• Theorem 4.10
• Lemma 4.11: Rate of Convergence
• Theorem 4.12: Asymptotic Normality
• Corollary 4.13
• Lemma C.1: Bin-Level Decomposition of Structure Matrix
• proof
• Lemma C.2: Norm of the weight vector in bin-space