Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases

TL;DR

Spawrious introduces a photorealistic, multi-environment benchmark for spurious correlations in image classification, explicitly modeling One-To-One and Many-To-Many relationships between backgrounds and classes across Easy, Medium, and Hard splits. It leverages a generation-pipeline (text-to-image plus image captioning) to create 152k labeled images, enabling controlled evaluation of group-robustness methods like ERM, GroupDRO, IRM, CORAL, CausIRL, and MMD-AAE. The study finds that state-of-the-art methods struggle, particularly on M2M-Hard splits, and that misclassifications are strongly tied to background cues, highlighting the need for methods capable of disentangling background bias from true causal features. The work also analyzes architectural effects, showing that larger models do not guarantee robustness and that ViT-based backbones may underperform CNNs in this setting, motivating broader methodological and modal extensions. Overall, Spawrious provides a rigorous, scalable platform for diagnosing and mitigating spurious-correlation biases in robust learning.

Abstract

The problem of spurious correlations (SCs) arises when a classifier relies on non-predictive features that happen to be correlated with the labels in the training data. For example, a classifier may misclassify dog breeds based on the background of dog images. This happens when the backgrounds are correlated with other breeds in the training data, leading to misclassifications during test time. Previous SC benchmark datasets suffer from varying issues, e.g., over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many (M2M) SCs arising between groups of spurious attributes and classes. In this paper, we present \benchmark-\{O2O, M2M\}-\{Easy, Medium, Hard\}, an image classification benchmark suite containing spurious correlations between classes and backgrounds. To create this dataset, we employ a text-to-image model to generate photo-realistic images and an image captioning model to filter out unsuitable ones. The resulting dataset is of high quality and contains approximately 152k images. Our experimental results demonstrate that state-of-the-art group robustness methods struggle with \benchmark, most notably on the Hard-splits with none of them getting over $70\%$ accuracy on the hardest split using a ResNet50 pretrained on ImageNet. By examining model misclassifications, we detect reliances on spurious backgrounds, demonstrating that our dataset provides a significant challenge.

Figures (5)

• Figure 1: Spawrious Challenges: Letters on the images denote the background, and the bottom bar in \ref{['fig:o2o']} indicates each class's proportion of the spurious background. In the O2O challenge, each class associates with a background during training, while the test data contains unseen combinations of class-background pairs. In the M2M challenge, a group of classes correlates with a group of backgrounds during training, but this correlation is reversed in the test data.
• Figure 2: Data distributions for our challenges:${\bm{x}}_i$ is a random image sampled, each $s_i$ is a spurious attribute, and each $c_i$ is a class label. The edges indicate the probability that the sample ${\bm{x}}_i$ has a given property, conditional on previous steps in the tree. The leaf nodes indicate the possible attribute-class combinations in the distribution. The colors emphasize the distribution shift in the test data.
• Figure 3: Spawrious Pipeline: We leverage text-to-image models for data generation and image-to-text models for cleaning.
• Figure 4: ERM misclassifications due to spurious correlations. The shown test images correspond to the class "Bulldog" with spurious backgrounds "Mountains" in the O2O-Hard (left) and "Snow" in the M2M-Hard (right) challenge.
• Figure 5: Confusion matrices for ERM models.$X$-axis: predictions; $Y$-axis: true labels.