Provably Accurate Shapley Value Estimation via Leverage Score Sampling

TL;DR

This paper addresses the computational bottleneck of Shapley value estimation for feature attribution in complex models, where exact computation costs grow as $O(2^n)$. It introduces Leverage SHAP, a near-linear-sample method that leverages leverage-score sampling to solve the associated regression problem with $O(n \log n)$ evaluations in expectation and strong non-asymptotic guarantees: with probability $1-\delta$, $\| \mathbf{Z}\tilde{\boldsymbol{\phi}} - \mathbf{y} \|_2^2 \le (1+\epsilon)\| \mathbf{Z}\boldsymbol{\phi}-\mathbf{y} \|_2^2$. A key theoretical contribution is the closed-form leverage scores $\ell_{S}=\binom{n}{|S|}^{-1}$, enabling efficient, unbiased sampling of subsets by size. The algorithm integrates paired sampling and sampling without replacement, achieving accurate estimates with $m=O(n \log(n/\delta)+ n/(\epsilon\delta))$ value-function evaluations and $O(mn^2)$ regression time. Empirically, Leverage SHAP consistently outperforms the optimized Kernel SHAP implementation across eight datasets, particularly when $n$ is large or the set-function $v$ is noisy, highlighting its practical impact for explainable AI and robust feature attribution.

Abstract

Originally introduced in game theory, Shapley values have emerged as a central tool in explainable machine learning, where they are used to attribute model predictions to specific input features. However, computing Shapley values exactly is expensive: for a general model with $n$ features, $O(2^n)$ model evaluations are necessary. To address this issue, approximation algorithms are widely used. One of the most popular is the Kernel SHAP algorithm, which is model agnostic and remarkably effective in practice. However, to the best of our knowledge, Kernel SHAP has no strong non-asymptotic complexity guarantees. We address this issue by introducing Leverage SHAP, a light-weight modification of Kernel SHAP that provides provably accurate Shapley value estimates with just $O(n\log n)$ model evaluations. Our approach takes advantage of a connection between Shapley value estimation and agnostic active learning by employing leverage score sampling, a powerful regression tool. Beyond theoretical guarantees, we show that Leverage SHAP consistently outperforms even the highly optimized implementation of Kernel SHAP available in the ubiquitous SHAP library [Lundberg & Lee, 2017].

Figures (12)

• Figure 1: Predicted versus true Shapley values for all features in 8 datasets (we use $m=5n$ samples for this experiment). Points near the identity line indicate that the estimated Shapley value is close to its true value. The plots suggest that our Leverage SHAP method is more accurate than the baseline Kernel SHAP algorithm, as well as the optimized Kernel SHAP implementation available in the SHAP library. We corroborate these findings with more experiments in Section \ref{['sec:experiments']}.
• Figure 2: Let $S$ be a subset sampled with the Kernel SHAP or Leverage SHAP probabilities. The plots show the distribution of the set size $|S|$ for different $n$. As in in the definition of Shapley values, the leverage score distribution places equal total weight on each subset size, contrasting with Kernel SHAP which over-weights small and large subsets.
• Figure 3: The $\ell_2$-norm error between the estimated Shapley values and ground truth Shapley values as a function of sample size. The lines report the median error while the shaded regions encompass the first to third quartile over 100 runs. Besides the setting where the exact Shapley values can be recovered, Leverage SHAP reliably outperforms even Optimized Kernel SHAP: the third quartile of the Leverage SHAP error is roughly the median error of Optimized Kernel SHAP.
• Figure 4: The $\ell_2$-norm error between the estimated and ground truth Shapley values as a function of noise in the set function. Leverage SHAP gives the best performance in all settings; the third quartile of the Leverage SHAP generally matches the median of the Optimized Kernel SHAP error.
• Figure 5: The $\ell_2$-norm error as a function of $\gamma$, the problem-specific parameter in Corollary \ref{['coro:l2norm']}. The lines indicate the median and the shaded regions encompass the first and third quartile over 100 runs. All three of the main algorithms we consider have higher $\ell_2$-norm error as $\gamma$ grows, suggesting that $\gamma$ is not an artifact of our analysis.

Theorems & Definitions (25)

• Theorem 1.1
• Lemma 2.0: Equivalence lundberg2017unifiedcharnes1988extremal
• Definition 2.1: Leverage Scores
• Lemma 3.0: Constrained to Unconstrained
• Lemma 3.1
• Lemma 3.2
• proof : Proof of Lemma \ref{['lemma:ATA_form']}
• proof : Proof of Lemma \ref{['lemma:leverage_scores']}
• Corollary 4.0
• Theorem A.1