Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

Thackshanaramana B

Abstract

The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.

Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

Abstract

The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.
Paper Structure (56 sections, 6 theorems, 23 equations, 1 figure, 5 tables)

This paper contains 56 sections, 6 theorems, 23 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Cross-method disagreement (same model).
  5. Within-class retraining noise.
  6. Rashomon sets and predictive multiplicity.
  7. XAI evaluation benchmarks.
  8. Actionability and recourse.
  9. Problem Formalization
  10. Relationship to the Rashomon Set.
  11. Theoretical Foundations
  12. Main Result
  13. Proof Strategy
  14. Step 1: SHAP decomposition by hypothesis class.
  15. Step 2: Within-class convergence.
  16. ...and 41 more sections

Key Result

Proposition 1

$\mathcal{L}_{\epsilon,\tau}(\mathbf{x}) \subsetneq \mathcal{R}_\epsilon \times \mathcal{R}_\epsilon$, where $\mathcal{L}_{\epsilon,\tau}(\mathbf{x}) = \{(M_1, M_2) \in \mathcal{R}_\epsilon^2 : M_1(\mathbf{x}) = M_2(\mathbf{x}) \;\land\; \rho(\boldsymbol{\varphi}_{M_1}, \boldsymbol{\varphi}_{M_2}) <

Figures (1)

  • Figure 1: Distribution of pairwise Spearman $\rho$ under the controlled same-split experiment. Within-class pairs (XGB vs. XGB, LR vs. LR) concentrate at $\rho = 1.000$; cross-class pairs (XGB vs. LR) spread across the full range with median $\rho = 0.364$, well below the lottery threshold ($\tau = 0.5$, dashed line). The structural gap $\Delta = 0.636$ directly validates Theorem \ref{['thm:main']} (Claims 1--2): the Agreement Gap is bounded away from zero and persists when training variance is eliminated.

Theorems & Definitions (15)

  • Definition 1: Prediction Equivalence
  • Definition 2: Explanation Disagreement
  • Definition 3: The Explanation Lottery
  • Definition 4: Lottery Rate
  • Proposition 1: Lottery is a Strict Subset of Rashomon
  • Theorem 1: Explanation Divergence Characterization
  • Lemma 1: Linear SHAP Collapse
  • Lemma 2: Tree SHAP Interaction Attribution
  • Lemma 3: Within-Class SHAP Convergence
  • Lemma 4: Cross-Class Attribution Bound
  • ...and 5 more