Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ShapBPT: Image Feature Attributions Using Data-Aware Binary Partition Trees

Muhammad Rashid, Elvio G. Amparore, Enrico Ferrari, Damiano Verda

TL;DR

ShapBPT introduces a data-aware, model-agnostic explainability method for vision that combines the Owen Shapley values with a Binary Partition Tree (BPT) to produce multiscale, morphologically aligned attributions. It leverages a data-driven partitioning strategy to reduce the number of model evaluations while improving localization of informative regions. Across diverse datasets and models, ShapBPT outperforms axis-aligned and segmentation-based baselines on AU-IoU and related metrics, and a 20-subject study shows human preferences for its explanations. By uniting hierarchical Shapley theory with image-aware hierarchies, the approach enables fast, semantically meaningful explanations for both CNNs and Vision Transformers.

Abstract

Pixel-level feature attributions are an important tool in eXplainable AI for Computer Vision (XCV), providing visual insights into how image features influence model predictions. The Owen formula for hierarchical Shapley values has been widely used to interpret machine learning (ML) models and their learned representations. However, existing hierarchical Shapley approaches do not exploit the multiscale structure of image data, leading to slow convergence and weak alignment with the actual morphological features. Moreover, no prior Shapley method has leveraged data-aware hierarchies for Computer Vision tasks, leaving a gap in model interpretability of structured visual data. To address this, this paper introduces ShapBPT, a novel data-aware XCV method based on the hierarchical Shapley formula. ShapBPT assigns Shapley coefficients to a multiscale hierarchical structure tailored for images, the Binary Partition Tree (BPT). By using this data-aware hierarchical partitioning, ShapBPT ensures that feature attributions align with intrinsic image morphology, effectively prioritizing relevant regions while reducing computational overhead. This advancement connects hierarchical Shapley methods with image data, providing a more efficient and semantically meaningful approach to visual interpretability. Experimental results confirm ShapBPT's effectiveness, demonstrating superior alignment with image structures and improved efficiency over existing XCV methods, and a 20-subject user study confirming that ShapBPT explanations are preferred by humans.

ShapBPT: Image Feature Attributions Using Data-Aware Binary Partition Trees

TL;DR

ShapBPT introduces a data-aware, model-agnostic explainability method for vision that combines the Owen Shapley values with a Binary Partition Tree (BPT) to produce multiscale, morphologically aligned attributions. It leverages a data-driven partitioning strategy to reduce the number of model evaluations while improving localization of informative regions. Across diverse datasets and models, ShapBPT outperforms axis-aligned and segmentation-based baselines on AU-IoU and related metrics, and a 20-subject study shows human preferences for its explanations. By uniting hierarchical Shapley theory with image-aware hierarchies, the approach enables fast, semantically meaningful explanations for both CNNs and Vision Transformers.

Abstract

Pixel-level feature attributions are an important tool in eXplainable AI for Computer Vision (XCV), providing visual insights into how image features influence model predictions. The Owen formula for hierarchical Shapley values has been widely used to interpret machine learning (ML) models and their learned representations. However, existing hierarchical Shapley approaches do not exploit the multiscale structure of image data, leading to slow convergence and weak alignment with the actual morphological features. Moreover, no prior Shapley method has leveraged data-aware hierarchies for Computer Vision tasks, leaving a gap in model interpretability of structured visual data. To address this, this paper introduces ShapBPT, a novel data-aware XCV method based on the hierarchical Shapley formula. ShapBPT assigns Shapley coefficients to a multiscale hierarchical structure tailored for images, the Binary Partition Tree (BPT). By using this data-aware hierarchical partitioning, ShapBPT ensures that feature attributions align with intrinsic image morphology, effectively prioritizing relevant regions while reducing computational overhead. This advancement connects hierarchical Shapley methods with image data, providing a more efficient and semantically meaningful approach to visual interpretability. Experimental results confirm ShapBPT's effectiveness, demonstrating superior alignment with image structures and improved efficiency over existing XCV methods, and a 20-subject user study confirming that ShapBPT explanations are preferred by humans.
Paper Structure (39 sections, 1 theorem, 18 equations, 23 figures, 2 tables, 3 algorithms)

This paper contains 39 sections, 1 theorem, 18 equations, 23 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Shapley values.
  4. Hierarchical coalition structures (HCS).
  5. The Owen approximation for Binary HCS.
  6. Hierarchical Coalition Structures for Images
  7. Generating BPT hierarchies.
  8. Experimental Assessment
  9. Comparison scores.
  10. Compared methods.
  11. Experiments.
  12. Numerical results.
  13. E8 - User preference study.
  14. Discussion and Other Related Works
  15. Conclusions
  16. ...and 24 more sections

Key Result

Theorem 1

Computational cost. Consider a BHCS consisting of a balanced tree of depth $d$. The time complexity of Eq. eq:BPT is in the order of $O(4^d)$ evaluations of the $\nu$ function.

Figures (23)

  • Figure 1: AA and BPT coalition structures for a sample image classification using a ResNet50 model.
  • Figure 2: (A) BPT generating by bottom-up merging coalitions from the pixels (1--6) to the root (11). (B) Details of one merging step $T_8{\downarrow}=\{T_4, T_5\}$ on some arbitrary coalition structure.
  • Figure 3: Shapley values for AA and BPT coalition structures, for different values of the budget $b$.
  • Figure 4: Selected saliency maps from experiments E1--E7 (summarized in Table \ref{['tab:summary']}) for various computer vision ML tasks.
  • Figure 5: Results for all scores across the E1--E7 experiments, with methods (on Y axis) ranked by performance (top to bottom).
  • ...and 18 more figures

Theorems & Definitions (6)

  • Theorem 1
  • proof
  • Example 1
  • Example 2
  • Example 3
  • Example 4