Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Saliency Maps are Ambiguous: Analysis of Logical Relations on First and Second Order Attributions

Leonid Schwenke, Martin Atzmueller

TL;DR

This work interrogates the reliability of saliency maps by showing that many attribution scores fail to capture all necessary classification information across diverse logical settings. It extends previous findings by introducing a Global Coherence Representation (GCR) to enable actual input omission and by incorporating second-order attributions via FCAM in a broad ANDOR-based experimental framework. The study demonstrates pervasive limitations across methods, with IntegratedGradients, LRP-Rollout, and Attention performing relatively best but still lacking universal robustness, especially in XOR-like scenarios. GCR provides a weight-based, global interpretation that helps diagnose when and how saliency scores truly reflect class-relevant information, offering a practical avenue for more faithful explanations and a foundation for future improvements.

Abstract

Recent work uncovered potential flaws in \eg attribution or heatmap based saliency methods. A typical flaw is a confirmations bias, where the scores are compared to human expectation. Since measuring the quality of saliency methods is hard due to missing ground truth model reasoning, finding general limitations is also hard. This is further complicated, because masking-based evaluation on complex data can easily introduce a bias, as most methods cannot fully ignore inputs. In this work, we extend our previous analysis on the logical dataset framework ANDOR, where we showed that all analysed saliency methods fail to grasp all needed classification information for all possible scenarios. Specifically, this paper extends our previous work using analysis on more datasets, in order to better understand in which scenarios the saliency methods fail. Further, we apply the Global Coherence Representation as an additional evaluation method in order to enable actual input omission.

Saliency Maps are Ambiguous: Analysis of Logical Relations on First and Second Order Attributions

TL;DR

This work interrogates the reliability of saliency maps by showing that many attribution scores fail to capture all necessary classification information across diverse logical settings. It extends previous findings by introducing a Global Coherence Representation (GCR) to enable actual input omission and by incorporating second-order attributions via FCAM in a broad ANDOR-based experimental framework. The study demonstrates pervasive limitations across methods, with IntegratedGradients, LRP-Rollout, and Attention performing relatively best but still lacking universal robustness, especially in XOR-like scenarios. GCR provides a weight-based, global interpretation that helps diagnose when and how saliency scores truly reflect class-relevant information, offering a practical avenue for more faithful explanations and a foundation for future improvements.

Abstract

Recent work uncovered potential flaws in \eg attribution or heatmap based saliency methods. A typical flaw is a confirmations bias, where the scores are compared to human expectation. Since measuring the quality of saliency methods is hard due to missing ground truth model reasoning, finding general limitations is also hard. This is further complicated, because masking-based evaluation on complex data can easily introduce a bias, as most methods cannot fully ignore inputs. In this work, we extend our previous analysis on the logical dataset framework ANDOR, where we showed that all analysed saliency methods fail to grasp all needed classification information for all possible scenarios. Specifically, this paper extends our previous work using analysis on more datasets, in order to better understand in which scenarios the saliency methods fail. Further, we apply the Global Coherence Representation as an additional evaluation method in order to enable actual input omission.
Paper Structure (36 sections, 4 equations, 21 figures, 5 tables)

This paper contains 36 sections, 4 equations, 21 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background and Assumptions
  4. Information Flow
  5. Background and Challenges on Saliency Maps Evaluation
  6. Redundancy
  7. Non-Informative Baseline
  8. Linearity
  9. Global Comparability
  10. Ablating Inputs
  11. Assumptions on Saliency Maps
  12. GCR
  13. GCR Classification and Membership Function
  14. Upscaling 1D to 2D
  15. Datasets
  16. ...and 21 more sections

Figures (21)

  • Figure 1: The process pipeline of our Global Coherence Representation, cf. schwenke2023extracting.
  • Figure 2: Example visualisation for a SAX discretization, cf. atzmueller2017explanation: Each data point from the original time series is mapped to a discrete symbol (a, b, c) based on the quantiles from the standard normal distribution.
  • Figure 3: Two FCAMs from the Synthetic dataset, w.r.t. class 4 (left) representing a slowly falling trend and class 6 (right) representing a sudden falling trend, cf. SA:21:globalschwenke2023extracting.
  • Figure 4: Two GTMs from the Synthetic dataset, w.r.t. class 4 (left) representing a slowly falling trend and class 6 (right) representing a sudden falling trend, cf. SA:21:globalschwenke2023extracting.
  • Figure 5: Framework for the ANDOR dataset.
  • ...and 16 more figures