Feature salience -- not task-informativeness -- drives machine learning model explanations

TL;DR

This study questions the common claim that feature attributions reveal what models have learned by showing that explanations are largely driven by test-time image salience rather than task informativeness. Using two controlled image-manipulation variants (watermarks and lightness) across confounded, balanced, and baseline training settings, the authors demonstrate that attribution methods consistently assign high importance to salient test-time features (RIW and RIL), often independent of the model's learned causal structure. Inverted colour encoding and SVD analyses further indicate that attribution maps track raw intensity and edge-like structures rather than genuine predictive features. The findings urge reevaluating prior conclusions about XAI effectiveness for detecting shortcut learning and call for integrating causal inference and robust benchmarks when interpreting feature attributions. Practically, the work cautions against using attribution maps for model debugging or data diagnostics without considering low-level image salience and distribution shifts.

Abstract

Explainable AI (XAI) promises to provide insight into machine learning models' decision processes, where one goal is to identify failures such as shortcut learning. This promise relies on the field's assumption that input features marked as important by an XAI must contain information about the target variable. However, it is unclear whether informativeness is indeed the main driver of importance attribution in practice, or if other data properties such as statistical suppression, novelty at test-time, or high feature salience substantially contribute. To clarify this, we trained deep learning models on three variants of a binary image classification task, in which translucent watermarks are either absent, act as class-dependent confounds, or represent class-independent noise. Results for five popular attribution methods show substantially elevated relative importance in watermarked areas (RIW) for all models regardless of the training setting ($R^2 \geq .45$). By contrast, whether the presence of watermarks is class-dependent or not only has a marginal effect on RIW ($R^2 \leq .03$), despite a clear impact impact on model performance and generalisation ability. XAI methods show similar behaviour to model-agnostic edge detection filters and attribute substantially less importance to watermarks when bright image intensities are encoded by smaller instead of larger feature values. These results indicate that importance attribution is most strongly driven by the salience of image structures at test time rather than statistical associations learned by machine learning models. Previous studies demonstrating successful XAI application should be reevaluated with respect to a possibly spurious concurrency of feature salience and informativeness, and workflows using feature attribution methods as building blocks should be scrutinised.

Figures (7)

• Figure 1: Generated images used and attribution maps obtained in the watermark study. A: Examples of images of cats and dogs with and without a watermark overlaid at a fixed position. Depending on the experimental setting studied, the watermark's prevalence is either higher for dogs than for cats (confounded setting) or not (balanced setting), or the watermark is not present at all (no-watermark setting) during training. B: Absolute-valued importance attributed to each pixel of a given test image by different feature attribution methods (Deconvolution, Integrated Gradients, Gradient SHAP, LRP-$\epsilon$, and LRP-$\alpha \beta$) and a model-independent edge-detection filter (discrete 2D Laplace filter). Shown are triplets of columns for models trained in the three settings and applied to test images with (WM) or without (No-WM) the watermark present, where the difference ($\Delta$) between the two attributions is also shown. Explanations obtained by feature attributions resemble results obtained by the model-independent discrete 2D Laplace filter and consistently highlight any present watermark irrespective of whether watermarks introduced confounding during model training or were balanced across cat and dog classes. Even for a model trained on data with no watermarks, substantial importance is attributed to watermarks in test images.
• Figure 2: Distributions of the relative importance on watermarks (RIW), quantifying the amount of importance attributed to pixels manipulated by insertion of a fixed-position translucent watermark in the watermark study, relative to the entire image. RIW distributions are shown for identical test images with and without the actual watermark applied, for different machine learning models (shown across rows), for which watermarks acted either as confounders (confounded setting), non-informative features (balanced setting) during training, or were completely absent during training (no-watermark setting). This is shown for a selection of feature attribution methods (Deconvolution, Integrated Gradients, Gradient SHAP, LRP-$\epsilon$, and LRP-$\alpha \beta$) as well as two model-independent baselines (Laplace and raw images $\mathbf{x}$), shown across columns. It can be seen that pixels manipulated by the watermark are attributed substantially higher RIW if a watermark is actually present, compared to the case where no watermark is present. This is true across all studied attribution methods and baselines but also across all experimental settings including the balanced and no-watermark settings, in which the presence of a watermark is non-informative about the class label.
• Figure 3: Examples of different lightness manipulations applied to the lightness channel of an animal image (upper row) in the lightness study, where the histogram of the lightness distribution is transformed to match a given beta distribution (lower row). Beta distributions $\text{Beta}(2,4)$ and $\text{Beta}(4, 2)$ were imposed on 50% of the images either dependent (confounded setting) or independent (balanced setting) of membership in one of two studied classes (animals and vehicles). In all non-manipulated images as well as all images in the baseline setting, lightness levels are normalised to a centred $\text{Beta}(3,3)$ distribution.
• Figure 4: Distributions of the relative importance on lightness (RIL), quantifying the amount of importance attributed to the lightness relative to all three channels of HLS-encoded animal and vehicle images in the lightness study. RIL distributions are shown for identical test images with lightness channels either normalised to a Beta$(3,3)$ distribution, darkened according to a Beta$(2,4)$ distribution or brightened according to a Beta$(4,2)$ distribution, for different machine learning models (shown across rows), for which lightness manipulations either induced confounding (confounded setting), were applied independent of class membership (balanced setting) during training, or were completely absent during training (baseline setting), and for a selection of feature attribution methods (Deconvolution, Integrated Gradients, Gradient SHAP, LRP-$\epsilon$, and LRP-$\alpha \beta$) as well as a model-independent baseline (raw images $\mathbf{x}$), shown across columns. The importance attributed to the lightness channel is strongly determined by the lightness of the test images with brightened images receiving higher importance than normalised baseline images, and baseline images receiving higher importance than darkened images. In contrast, there are no notable changes in attribution distributions for models trained in the confounded, balanced, or baseline settings.
• Figure S1: Absolute-valued first singular vectors of pixel $\times$ sample attribution matrices produced for cat and dog test images with watermarks in the watermark study using models trained on confounded, balanced, and no-watermark data (shown across rows) in combination with five different feature attribution methods (Deconvolution, Integrated Gradients, Gradient SHAP, LRP-$\epsilon$, and LRP-$\alpha \beta$) and two model-independent baselines ((Laplace and raw images $\mathbf{x}$, shown across columns). The watermark is clearly visible for every combination of model training setting and attribution method, confirming the watermark as the most salient structure in all studied cases including cases where watermarks were present during training or represented non-informative nuisance features to the model. Singular vectors obtained from attributions closely resemble singular vectors obtained from model-independent edge-detection filters (Laplace) and even the raw input image data ($\mathbf{x}$).