Semantic Prioritization in Visual Counterfactual Explanations with Weighted Segmentation and Auto-Adaptive Region Selection

TL;DR

This work tackles the interpretability gap in fine-grained visual counterfactual explanations by introducing WSAR-Net, a non-generative CE framework that constrains edits to semantically relevant regions and optimizes the editing order. It combines a weighted semantic map, derived from attribution and segmentation, with an auto-adaptive candidate editing sequence to minimize editing of non-semantic units while accelerating class transitions. Key contributions include the weighted semantic map formulation $M_{sem}=S_q\circ M_c$, the sparse editing mechanism via $f(I^*)=(\mathds{1}-\mathbf{a})\circ f(I)+\mathbf{a}\circ P f(I')$, and the dual-stage optimization that scales to multiple distractors through a similarity-guided pruning of edit permutations. Experiments on CUB-200-2011 and Stanford Dogs with ResNet-50 and VGG-16 demonstrate improved semantic coherence (Near-KP, Same-KP) and reduced editing effort, underscoring practical gains in efficient, interpretable counterfactual explanations.

Abstract

In the domain of non-generative visual counterfactual explanations (CE), traditional techniques frequently involve the substitution of sections within a query image with corresponding sections from distractor images. Such methods have historically overlooked the semantic relevance of the replacement regions to the target object, thereby impairing the model's interpretability and hindering the editing workflow. Addressing these challenges, the present study introduces an innovative methodology named as Weighted Semantic Map with Auto-adaptive Candidate Editing Network (WSAE-Net). Characterized by two significant advancements: the determination of an weighted semantic map and the auto-adaptive candidate editing sequence. First, the generation of the weighted semantic map is designed to maximize the reduction of non-semantic feature units that need to be computed, thereby optimizing computational efficiency. Second, the auto-adaptive candidate editing sequences are designed to determine the optimal computational order among the feature units to be processed, thereby ensuring the efficient generation of counterfactuals while maintaining the semantic relevance of the replacement feature units to the target object. Through comprehensive experimentation, our methodology demonstrates superior performance, contributing to a more lucid and in-depth understanding of visual counterfactual explanations.

Figures (8)

• Figure 1: (a) Counterfactual visual explanations method based on region-editing replacement. (b) Compared with previous work 14, our method demonstrates greater semantic relevance to the target object.
• Figure 2: Exhaustive editing of feature units: sequentially replacing all cells of the distractor with the cell at position indices 0 (a) and 1 (b) of the queries and predicted by classifier g.
• Figure 3: Overview of the proposed WSAR-Net framework. It consists of three key components: feature extraction $f$ and a segmentation model $f_{S}$ to create weighted semantic maps, and sequence optimization during editing. By integrating the model's attribution map with segmentation matrices, we establish a weighted semantic map for the query. Simultaneously, the semantic region is defined based on the segmentation matrix from the distractor image. In the auto-adaptive candidate editing phase, we filter for the top-$m$ units by weight scores in the weighted semantic map, then compute similarity against the semantic features of the distractor image to derive similarity scores. From these, the top-$k$ units with the highest similarity are combined with the query image's $m$ units with the highest weight score. Subsequent replacements in the query's feature map based on these combinations lead to recalculating the counterfactual class until a change in the predicted class transition occurs. $L_\text{sim}$ and $L_\text{cls}$ represent the similarity and classification losses, respectively.
• Figure 4: Visualization of computed counterfactuals. (a) Each query image computes the counterfactual using the feature region from a single distractor image. The area outlined in red represents the region whose replacement into the query image results in a class transition. (b) In the process of selecting units from multiple distractor images for continuous replacement, the distractor image within the yellow area eventually becomes the one containing the replaced unit that leads to the query class transition.
• Figure 5: Validation of efficiency for counterfactual generation. (a) Class probability comparison of randomly selected (40) counterfactuals with the baseline method during the generation process on two datasets with ResNet-50 and VGG-16. (b) Comparative analysis of the time distribution for generation and the distribution of class probabilities among randomly selected (40) counterfactuals relative to the baseline method on two datasets with ResNet-50 and VGG-16. $(*)$ is our method, result of CUB-200-2011 dataset on left two columns, right two columns for Stanford Dogs dataset.