Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustness

TL;DR

The paper challenges the belief that higher shape bias causally boosts OOD robustness. It evaluates 39 augmentations on ImageNet-100 and tests on a diverse OOD Benchmark (17 perturbations) using ResNet-50, quantifying both in-domain and OOD performance. The results show no consistent causal link between shape bias and OOD robustness and reveal ImageNet-1K biases that can be mitigated with augmentation. Importantly, some augmentations improve both in-domain accuracy and OOD robustness, offering practical guidance for augmentation choices.

Abstract

Deep learning models can perform well when evaluated on images from the same distribution as the training set. However, applying small perturbations in the forms of noise, artifacts, occlusions, blurring, etc. to a model's input image and feeding the model with out-of-distribution (OOD) data can significantly drop the model's accuracy, making it not applicable to real-world scenarios. Data augmentation is one of the well-practiced methods to improve model robustness against OOD data; however, examining which augmentation type to choose and how it affects the OOD robustness remains understudied. There is a growing belief that augmenting datasets using data augmentations that improve a model's bias to shape-based features rather than texture-based features results in increased OOD robustness for Convolutional Neural Networks trained on the ImageNet-1K dataset. This is usually stated as ``an increase in the model's shape bias results in an increase in its OOD robustness". Based on this hypothesis, some works in the literature aim to find augmentations with higher effects on model shape bias and use those for data augmentation. By evaluating 39 types of data augmentations on a widely used OOD dataset, we demonstrate the impact of each data augmentation on the model's robustness to OOD data and further show that the mentioned hypothesis is not true; an increase in shape bias does not necessarily result in higher OOD robustness. By analyzing the results, we also find some biases in the ImageNet-1K dataset that can easily be reduced using proper data augmentation. Our evaluation results further show that there is not necessarily a trade-off between in-domain accuracy and OOD robustness, and choosing the proper augmentations can help increase both in-domain accuracy and OOD robustness simultaneously.

Figures (4)

• Figure 1: This illustration shows how a ResNet-50 model with 89.06% in-domain accuracy fails when a small perturbation has been applied to the image. The label on top of each image is the name of the perturbation, and the label below each image is the predicted class. "reference" is the clean image.
• Figure 2: An overview of how different augmentations alter the reference (original) image. Best viewed in color.
• Figure 3: Evaluation of OOD robustness for various augmentations on the OOD Benchmark dataset. The pink bar represents the Vanilla (baseline with no data augmentations) model.
• Figure 4: This figure shows the relationship between the shape bias and the OOD robustness of models trained on augmented datasets. The evaluation has been performed on the OOD Benchmark dataset that contains two groups of datasets, called the Noise and the Style groups.