Focusing Image Generation to Mitigate Spurious Correlations

TL;DR

This work addresses the problem of spurious correlations that cause image classifiers to attend to background attributes rather than instance features. It introduces Spurious Correlations Guided Synthesis (SCGS), a data-augmentation framework that does not require group labels or a concept bank; it identifies misclassified samples, clusters them to capture diverse spurious patterns, creates masks via Grad-CAM++ on sampled misclassified images, and uses Stable Diffusion to generate new, balanced images that emphasize the object. Across MetaShift, Waterbirds, and CelebA, SCGS improves worst-group accuracy and, in combination with JTT, approaches or surpasses baseline methods that rely on group labels. The method also demonstrates that CAM-based masking yields higher debiasing efficacy than alternative masking strategies, and reveals through attention visualization that SCGS shifts focus toward instance features. Limitations appear on very large datasets, where the debiasing impact diminishes, pointing to opportunities for scaling and integration with additional debiasing techniques.

Abstract

Instance features in images exhibit spurious correlations with background features, affecting the training process of deep neural classifiers. This leads to insufficient attention to instance features by the classifier, resulting in erroneous classification outcomes. In this paper, we propose a data augmentation method called Spurious Correlations Guided Synthesis (SCGS) that mitigates spurious correlations through image generation model. This approach does not require expensive spurious attribute (group) labels for the training data and can be widely applied to other debiasing methods. Specifically, SCGS first identifies the incorrect attention regions of a pre-trained classifier on the training images, and then uses an image generation model to generate new training data based on these incorrect attended regions. SCGS increases the diversity and scale of the dataset to reduce the impact of spurious correlations on classifiers. Changes in the classifier's attention regions and experimental results on three different domain datasets demonstrate that this method is effective in reducing the classifier's reliance on spurious correlations.

Figures (3)

• Figure 1: (a) Existing DISCpaper9 method generates new images based on a concept bank and concept probabilities. (b) SCGS generates new images by using labels, masks, and misclassified images from training set as inputs for an existing image generation modelpaper6.
• Figure 2: Overview of our SCGS.
• Figure 3: Grad-CAM++ comparison of ERM and SCGS.