MNIST-C: A Robustness Benchmark for Computer Vision

TL;DR

Vision models often fail on out-of-distribution data; this paper proposes MNIST-C, a robustness benchmark consisting of 15 realistic corruptions applied to MNIST to probe OOD performance. It analyzes multiple models and shows substantial degradation and that adversarial defenses can worsen MNIST-C robustness, underscoring gaps left by adversarial-robustness measures. It demonstrates that naive data augmentation or training on corruption distributions does not solve the task and advocates MNIST-C as a reliable, reproducible tool for evaluating robustness. The authors release the corruption suite and code to enable broad benchmarking and drive development of robust feature representations for real-world vision systems.

Abstract

We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. Through several experiments and visualizations we demonstrate that our corruptions significantly degrade performance of state-of-the-art computer vision models while preserving the semantic content of the test images. In contrast to the popular notion of adversarial robustness, our model-agnostic corruptions do not seek worst-case performance but are instead designed to be broad and diverse, capturing multiple failure modes of modern models. In fact, we find that several previously published adversarial defenses significantly degrade robustness as measured by MNIST-C. We hope that our benchmark serves as a useful tool for future work in designing systems that are able to learn robust feature representations that capture the underlying semantics of the input.

Figures (8)

• Figure 1: Randomly sampled applications of all 15 corruptions comprising MNIST-C
• Figure 2: Mean test accuracy and relative mean Corruption Error (mCE) of various models on MNIST-C. Best performance is bolded.
• Figure 3: Test accuracy of various models on MNIST-C. Highest accuracy per row is shaded green, lowest accuracy is shaded red, and average accuracy is shaded white.
• Figure 4: Test accuracy of training Conv1 with simple data augmentation methods on MNIST-C. Clean refers to no augmentation, single trains and tests on a single corruption, all-but-one trains on all 31 but the tested corruption, and all trains on all 31 corruptions.
• Figure 5: Randomly sampled test errors by Conv1 on brightness, Canny edges, dotted line, and fog.