Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XIMAGENET-12: An Explainable AI Benchmark Dataset for Model Robustness Evaluation

Qiang Li, Dan Zhang, Shengzhao Lei, Xun Zhao, Porawit Kamnoedboon, WeiWei Li, Junhao Dong, Shuyan Li

TL;DR

XIMAGENET-12 introduces an explainable robustness benchmark for vision models by combining precise foreground/background annotations with six realistic perturbations across 12 ImageNet categories. It defines a variance-based robustness score that jointly accounts for cross-scenario and within-scenario performance, and demonstrates how background manipulation affects state-of-the-art backbones and segmentation models. The results reveal that background changes, especially random substitutions, can drastically reduce accuracy, while models trained on segmented foregrounds show resilience to missing backgrounds; importantly, higher nominal accuracy does not always correlate with robustness. The dataset and accompanying code enable detailed cross-scenario robustness evaluation, with practical implications for deployment in real-world settings and for guiding model selection in industrial applications and domain adaptation tasks.

Abstract

Despite the promising performance of existing visual models on public benchmarks, the critical assessment of their robustness for real-world applications remains an ongoing challenge. To bridge this gap, we propose an explainable visual dataset, XIMAGENET-12, to evaluate the robustness of visual models. XIMAGENET-12 consists of over 200K images with 15,410 manual semantic annotations. Specifically, we deliberately selected 12 categories from ImageNet, representing objects commonly encountered in practical life. To simulate real-world situations, we incorporated six diverse scenarios, such as overexposure, blurring, and color changes, etc. We further develop a quantitative criterion for robustness assessment, allowing for a nuanced understanding of how visual models perform under varying conditions, notably in relation to the background. We make the XIMAGENET-12 dataset and its corresponding code openly accessible at \url{https://sites.google.com/view/ximagenet-12/home}. We expect the introduction of the XIMAGENET-12 dataset will empower researchers to thoroughly evaluate the robustness of their visual models under challenging conditions.

XIMAGENET-12: An Explainable AI Benchmark Dataset for Model Robustness Evaluation

TL;DR

XIMAGENET-12 introduces an explainable robustness benchmark for vision models by combining precise foreground/background annotations with six realistic perturbations across 12 ImageNet categories. It defines a variance-based robustness score that jointly accounts for cross-scenario and within-scenario performance, and demonstrates how background manipulation affects state-of-the-art backbones and segmentation models. The results reveal that background changes, especially random substitutions, can drastically reduce accuracy, while models trained on segmented foregrounds show resilience to missing backgrounds; importantly, higher nominal accuracy does not always correlate with robustness. The dataset and accompanying code enable detailed cross-scenario robustness evaluation, with practical implications for deployment in real-world settings and for guiding model selection in industrial applications and domain adaptation tasks.

Abstract

Despite the promising performance of existing visual models on public benchmarks, the critical assessment of their robustness for real-world applications remains an ongoing challenge. To bridge this gap, we propose an explainable visual dataset, XIMAGENET-12, to evaluate the robustness of visual models. XIMAGENET-12 consists of over 200K images with 15,410 manual semantic annotations. Specifically, we deliberately selected 12 categories from ImageNet, representing objects commonly encountered in practical life. To simulate real-world situations, we incorporated six diverse scenarios, such as overexposure, blurring, and color changes, etc. We further develop a quantitative criterion for robustness assessment, allowing for a nuanced understanding of how visual models perform under varying conditions, notably in relation to the background. We make the XIMAGENET-12 dataset and its corresponding code openly accessible at \url{https://sites.google.com/view/ximagenet-12/home}. We expect the introduction of the XIMAGENET-12 dataset will empower researchers to thoroughly evaluate the robustness of their visual models under challenging conditions.
Paper Structure (12 sections, 3 equations, 16 figures, 3 tables)

This paper contains 12 sections, 3 equations, 16 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. XIMAGENET-12 Dataset
  4. Dataset Simulation
  5. Dataset Properties
  6. Robustness Score
  7. Experiments
  8. Experimental Settings
  9. Main Results
  10. Qualitative Results
  11. DISCUSSION AND CONCLUSION
  12. Appendix

Figures (16)

  • Figure 1: XIMAGENET-12 sample for 6 scenarios: Blur, Random generated background, AI-generated background, Segmentated, Transparent and Color images. Over 200K Images in total.
  • Figure 2: The flow chart of XIMAGENET-12 generation.
  • Figure 3: XIMAGENET-12 samples for each class and scenario.
  • Figure 4: Saliency Map Analysis on the MVTec AD Dataset mvtec.
  • Figure 5: TOP-1 Accuracy for SOTA models pre-trained on IMAGENET original formal images and tested on XIMAGENET-12 different background scenarios.
  • ...and 11 more figures