Mitigating Clever Hans Strategies in Image Classifiers through Generating Counterexamples

TL;DR

This work tackles the problem of Clever Hans predictors caused by spurious correlations in image classifiers, including foundation models. It introduces Counterfactual Knowledge Distillation (CFKD), which uses a counterfactual explainer and teacher feedback to generate and label counterfactuals, enriching the training data and distilling robust decision boundaries without requiring confounder labels. CFKD demonstrates strong, balanced generalization across five datasets, with ablations showing the crucial role of the SCE counterfactual explainer and the choice of teacher. The approach scales to multiple confounders and performs well in low-data settings, offering practical impact for robust AI in safety-critical domains such as medical imaging and industrial quality control.

Abstract

Deep learning models remain vulnerable to spurious correlations, leading to so-called Clever Hans predictors that undermine robustness even in large-scale foundation and self-supervised models. Group distributional robustness methods, such as Deep Feature Reweighting (DFR) rely on explicit group labels to upweight underrepresented subgroups, but face key limitations: (1) group labels are often unavailable, (2) low within-group sample sizes hinder coverage of the subgroup distribution, and (3) performance degrades sharply when multiple spurious correlations fragment the data into even smaller groups. We propose Counterfactual Knowledge Distillation (CFKD), a framework that sidesteps these issues by generating diverse counterfactuals, enabling a human annotator to efficiently explore and correct the model's decision boundaries through a knowledge distillation step. Unlike DFR, our method not only reweights the undersampled groups, but it also enriches them with new data points. Our method does not require any confounder labels, achieves effective scaling to multiple confounders, and yields balanced generalization across groups. We demonstrate CFKD's efficacy across five datasets, spanning synthetic tasks to an industrial application, with particularly strong gains in low-data regimes with pronounced spurious correlations. Additionally, we provide an ablation study on the effect of the chosen counterfactual explainer and teacher model, highlighting their impact on robustness.

Figures (5)

• Figure 1: Proposed CFKD method compared to a subgroup reweighting baseline (e.g. DFR). In the shown 'blond' vs. 'non-blond' classification task, the attribute 'male/female' becomes a spurious correlate due to the low representation of 'blond males' in the data, leading to a Clever Hans effect. CFKD addresses this by populating underrepresented groups through counterfactual generation. Specifically, generating counterfactuals for 'blond females' occasionally results in 'blond males', revealing the CH effect. CFKD then relabels these misclassified instances and adds them to the dataset (step 1) followed by retraining (step 2). This guided data augmentation/labeling/retraining strategy allows CFKD to address CH effects more directly than subgroup reweighting.
• Figure 2: Cartoon depiction of the Follicle dataset and its spurious correlation. If the number of inner Granulosa cells is greater than or equal to one, a Follicle is considered to be of class 2 ('growing follicle'); otherwise, it is considered to be of class 1 ('primordial follicle'). The size of the follicle is, from a biological perspective, not determining the class, yet it is spuriously correlated to the number of Granulosa cells. Specifically, the dataset contains 1507 samples, with 776 small primordial (51.5% of the data), 459 big growing (30.5%), 155 big primordial (10.3%), and 117 small growing (7.8%).
• Figure 3: Effect of sample size and spurious correlation level on accuracy of benchmarked methods. Results are shown for the Square and the CelebA Smiling vs. Copyrighttag datasets. When analyzing the effect of sample size, we fix the correlation parameter to $0.96$. When analyzing the effect of correlation strength, we fix the sample size to $1000$. DFR and RR-ClArC results are not reported for a correlation parameter of $1.0$ as the methods are not applicable in that setting. For a correlation parameter of $0.0$, DFR reduces to the original training.
• Figure 4: A comparison between the different teachers that were used on the Square dataset with correlation parameter $0.6$. The performance is measured before and after applying one iteration of CFKD.
• Figure 5: Qualitative analysis of CFKD, where for a variety of data points from the datasets considered (top-row), we display one counterfactual generated before applying CFKD (middle row) and after applying CFKD (bottom row). These images highlight the transition from initially perturbing non-causal features (e.g. follicle size) to finally perturbing causal features (e.g. inner Granulosa cells).