How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations

TL;DR

This work reveals a surprising dependency of post-hoc explanations on how the classification head is trained, even when the backbone is frozen. By systematically probing frozen representations with linear and non-linear heads across supervised, self-supervised, and vision-language pretraining, the authors show that training objective and probe complexity drastically shape attribution quality. They identify Binary Cross-Entropy (BCE) losses and interpretable B-cos MLP probes as particularly effective for achieving class-specific, localized explanations, improving metrics like GridPG and EPG across CNNs and Vision Transformers. The findings have practical implications for XAI deployment, suggesting that tuning the probe can yield more faithful and interpretable explanations without retraining the entire backbone, across diverse pretraining regimes and explanation methods.

Abstract

Post-hoc importance attribution methods are a popular tool for "explaining" Deep Neural Networks (DNNs) and are inherently based on the assumption that the explanations can be applied independently of how the models were trained. Contrarily, in this work we bring forward empirical evidence that challenges this very notion. Surprisingly, we discover a strong dependency on and demonstrate that the training details of a pre-trained model's classification layer (less than 10 percent of model parameters) play a crucial role, much more than the pre-training scheme itself. This is of high practical relevance: (1) as techniques for pre-training models are becoming increasingly diverse, understanding the interplay between these techniques and attribution methods is critical; (2) it sheds light on an important yet overlooked assumption of post-hoc attribution methods which can drastically impact model explanations and how they are interpreted eventually. With this finding we also present simple yet effective adjustments to the classification layers, that can significantly enhance the quality of model explanations. We validate our findings across several visual pre-training frameworks (fully-supervised, self-supervised, contrastive vision-language training) and analyse how they impact explanations for a wide range of attribution methods on a diverse set of evaluation metrics.

Figures (42)

• Figure 1: Impact of Loss (BCE vs. CE). (a) EPG Scores, and (b) Pixel Deletion scores for Bcos and LRP attributions for a linear probe when trained on frozen pre-trained features (DINO in this case). We find that BCE probes lead to more localized and stable attributions, thus higlighting the significant impact of the loss function on well-established attribution methods. Interestingly, despite the fact that the models differ only by a single classification layer, the attributions show stark differences in commonly used metrics for evaluating their quality.
• Figure 2: Setup: Step 1. Linear or MLP probes $h$ are trained on frozen pre-trained models $f$. Step 2. Explanation methods are applied to the classification predictions of the trained probes, and evaluated across a wide array of interpretability metrics to assess explanation quality (e.g. localization).
• Figure 3: Due to the shift-invariance of softmax, one cannot expect positive and negative attributions to be well calibrated, which can lead to unintuitive model explanations, see also \ref{['eq:ce_shift_w']}. Specifically, one can easily define equivalent linear probes (Probe 1,2) that achieve the same CE loss, but visually dissimilar explanations and GridPG scores (65.7% vs. 11.9%). Col. 2+3 show LRP lrp attributions for two equivalent probes explaining the same class (bighorn).
• Figure 4: BCE vs. CE --- Accuracy and GridPG scores on ImageNet for ResNet-50 models. GridPG scores improve significantly with BCE loss over CE loss, and this is consistent across pre-training paradigms for both B-cos models (top row) and conventional models (bottom row).
• Figure 5: (a) BCE vs. CE. The B-cos attributions for a linear probe trained with CE loss (bottom row) leak into nearby cells in the 2x2 GridPG evaluation setting. The attributions for linear probes trained with a BCE loss (top row) are consistently much more localized. (b) SSL vs Supervised. The B-cos attributions for DINO (cols. 2+3) are visually very similar to supervised models (cols. 4+5), despite being optimized very differently, thus highlighting the importance of the training objective of the linear probe (see \ref{['supp:sec:qualitative:setting1']} for more results).