An XAI-based Analysis of Shortcut Learning in Neural Networks

TL;DR

The paper tackles shortcut learning by exposing how neural networks pick up spurious correlations that correlate with labels but lack causal basis. It introduces the neuron spurious score ($s$-score), an XAI-based diagnostic to quantify a neuron’s reliance on spurious features, and analyzes both CNNs and ViTs to reveal architecture-specific encoding and entanglement patterns. The findings show that spurious features shape the latent space and that some disentanglement is possible, especially in CNNs, but not uniformly across architectures; post-hoc methods like deep feature re-weighting (DFR) or targeted pruning improve minority-group performance by changing neuron interactions rather than truly removing spurious encodings. The work highlights limitations of existing mitigation approaches and argues for architecture-aware strategies that account for how spurious features are encoded across layers and attention heads, laying groundwork for safer, more robust AI systems.

Abstract

Machine learning models tend to learn spurious features - features that strongly correlate with target labels but are not causal. Existing approaches to mitigate models' dependence on spurious features work in some cases, but fail in others. In this paper, we systematically analyze how and where neural networks encode spurious correlations. We introduce the neuron spurious score, an XAI-based diagnostic measure to quantify a neuron's dependence on spurious features. We analyze both convolutional neural networks (CNNs) and vision transformers (ViTs) using architecture-specific methods. Our results show that spurious features are partially disentangled, but the degree of disentanglement varies across model architectures. Furthermore, we find that the assumptions behind existing mitigation methods are incomplete. Our results lay the groundwork for the development of novel methods to mitigate spurious correlations and make AI models safer to use in practice.

Figures (10)

• Figure 1: Examples of variety shortcut types including backgrounds, texture and artifacts. The leftmost image is from the Waterbirds dataset, the three center images are from Geirhos:2020:nature, and the rightmost image is from the ISIC dataset.
• Figure 2: Visualization of the $\mathbf{s}$-score derivation by examples on ISIC dataset (left) and Waterbirds (right). A binary mask of the spurious feature (color patch or background) is obtained from the annotations in the training data set (top row). We use GradCam to obtain a feature attribution heatmap from a neuron and intersect its binarized version with the segmentation map (bottom row).
• Figure 3: Visualization of representation of last layers. Clusters are mainly defined by the spurious attribute (patch for ISIC, land/water for birds) and not by the classes (malignant/benign for ISIC and landbird/waterbird for birds).
• Figure 4: Neuron heatmaps overlaid with original images, illustrating the activation of three different neurons (rows) across five sample images (columns).
• Figure 5: Average $\mathbf{s}$-score of ResNet18 and DFR trained on ISIC (left) and Waterbirds (right).