MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions

TL;DR

MedMNIST-C addresses the critical challenge of robustness and distribution shifts in medical imaging by introducing a unified, ImageNet-C-inspired benchmark spanning 12 MedMNIST+ datasets across 9 modalities with dataset-specific corruptions at five severity levels. The authors couple this benchmark with a corruption-API for targeted data augmentation, enabling domain-knowledge-inspired transformations during training. They evaluate multiple architectures and show that targeted augmentations yield superior robustness (higher $AUC$ gains) than generic methods, particularly for small datasets, and validate the approach via a rigorous, bias-m mitigating $k$-fold setup. Overall, MedMNIST-C provides a scalable, open framework that can drive development of robust medical image analysis methods and facilitate reproducible assessment of domain generalization under realistic corruptions, with metrics based on $BE$ and $rBE$ to quantify performance shifts. $BE$ and $rBE$ capture how distribution shifts degrade accuracy, guiding the design of more resilient models in clinical settings.

Abstract

The integration of neural-network-based systems into clinical practice is limited by challenges related to domain generalization and robustness. The computer vision community established benchmarks such as ImageNet-C as a fundamental prerequisite to measure progress towards those challenges. Similar datasets are largely absent in the medical imaging community which lacks a comprehensive benchmark that spans across imaging modalities and applications. To address this gap, we create and open-source MedMNIST-C, a benchmark dataset based on the MedMNIST+ collection covering 12 datasets and 9 imaging modalities. We simulate task and modality-specific image corruptions of varying severity to comprehensively evaluate the robustness of established algorithms against real-world artifacts and distribution shifts. We further provide quantitative evidence that our simple-to-use artificial corruptions allow for highly performant, lightweight data augmentation to enhance model robustness. Unlike traditional, generic augmentation strategies, our approach leverages domain knowledge, exhibiting significantly higher robustness when compared to widely adopted methods. By introducing MedMNIST-C and open-sourcing the corresponding library allowing for targeted data augmentations, we contribute to the development of increasingly robust methods tailored to the challenges of medical imaging. The code is available at https://github.com/francescodisalvo05/medmnistc-api .

Figures (1)

• Figure 1: Overview of four different corruptions applied (from top to bottom) to PathMNIST, ChestMNIST, DermaMNIST, and RetinaMNIST.