Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion

Abu Noman Md Sakib, OFM Riaz Rahman Aranya, Kevin Desai, Zijie Zhang

Abstract

Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in segmentation explainability. Code is available at https://github.com/anmspro/DEA.

Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion

Abstract

Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in segmentation explainability. Code is available at https://github.com/anmspro/DEA.
Paper Structure (17 sections, 5 equations, 4 figures, 1 table)

This paper contains 17 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Setup
  5. Dual-Evidence Attribution
  6. Metrics
  7. Experimental Setup
  8. Datasets and Models
  9. Settings
  10. Results
  11. Quantitative Results
  12. Qualitative Results
  13. Discussion
  14. Conclusion
  15. Appendix
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Overview of DEA. Elementwise gradient evidence (EGA) and region intervention evidence (RIA) are combined through multiplicative agreement and residual intervention support.
  • Figure 2: Representative success cases where DEA improves target-region faithfulness while preserving spatial focus.
  • Figure 3: Mechanistic decomposition of DEA: elementwise gradient map, region intervention map, interaction, and corrected output (single case).
  • Figure 4: Representative SBD failure cases under the same comparison pipeline used for main-text figures. Residual off-target activation remains in cluttered contexts, and fine boundary detail can be missed on thin target structures.