Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches

Maura Pintor, Daniele Angioni, Angelo Sotgiu, Luca Demetrio, Ambra Demontis, Battista Biggio, Fabio Roli

TL;DR

ImageNet-Patch introduces a fast, transferable adversarial patch benchmark by pre-optimizing patches on an ensemble of models and applying them to ImageNet images under random affine transforms. This yields a 50,000-sample dataset (10 patches × 5,000 images) to rapidly assess robustness across many architectures, including both standard and robustly trained models, with a physical-world validation showing consistent trends. The method highlights the value of transferability and affine invariance for rapid robustness evaluation, while also noting limitations vs full adversarial attacks. The work positions ImageNet-Patch as a complementary benchmark to robustness suites like RobustBench and as a foundation for cross-model leaderboard-style evaluation of patch-based robustness.

Abstract

Adversarial patches are optimized contiguous pixel blocks in an input image that cause a machine-learning model to misclassify it. However, their optimization is computationally demanding, and requires careful hyperparameter tuning, potentially leading to suboptimal robustness evaluations. To overcome these issues, we propose ImageNet-Patch, a dataset to benchmark machine-learning models against adversarial patches. It consists of a set of patches, optimized to generalize across different models, and readily applicable to ImageNet data after preprocessing them with affine transformations. This process enables an approximate yet faster robustness evaluation, leveraging the transferability of adversarial perturbations. We showcase the usefulness of this dataset by testing the effectiveness of the computed patches against 127 models. We conclude by discussing how our dataset could be used as a benchmark for robustness, and how our methodology can be generalized to other domains. We open source our dataset and evaluation code at https://github.com/pralab/ImageNet-Patch.

ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches

TL;DR

ImageNet-Patch introduces a fast, transferable adversarial patch benchmark by pre-optimizing patches on an ensemble of models and applying them to ImageNet images under random affine transforms. This yields a 50,000-sample dataset (10 patches × 5,000 images) to rapidly assess robustness across many architectures, including both standard and robustly trained models, with a physical-world validation showing consistent trends. The method highlights the value of transferability and affine invariance for rapid robustness evaluation, while also noting limitations vs full adversarial attacks. The work positions ImageNet-Patch as a complementary benchmark to robustness suites like RobustBench and as a foundation for cross-model leaderboard-style evaluation of patch-based robustness.

Abstract

Adversarial patches are optimized contiguous pixel blocks in an input image that cause a machine-learning model to misclassify it. However, their optimization is computationally demanding, and requires careful hyperparameter tuning, potentially leading to suboptimal robustness evaluations. To overcome these issues, we propose ImageNet-Patch, a dataset to benchmark machine-learning models against adversarial patches. It consists of a set of patches, optimized to generalize across different models, and readily applicable to ImageNet data after preprocessing them with affine transformations. This process enables an approximate yet faster robustness evaluation, leveraging the transferability of adversarial perturbations. We showcase the usefulness of this dataset by testing the effectiveness of the computed patches against 127 models. We conclude by discussing how our dataset could be used as a benchmark for robustness, and how our methodology can be generalized to other domains. We open source our dataset and evaluation code at https://github.com/pralab/ImageNet-Patch.
Paper Structure (5 sections, 2 equations, 4 figures, 1 algorithm)

This paper contains 5 sections, 2 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Crafting Transferable Adversarial Patches
  3. The ImageNet-Patch Dataset
  4. Experimental Analysis
  5. Evaluation Metrics

Figures (4)

  • Figure 1: The three-step methodology followed to build our ImageNet-Patch benchmark.
  • Figure 2: The optimization process, graphically described. At each step, we apply the patch to be optimized with random affine transformations on sample images, and we compute the scores of the ensemble. Hence, the algorithm computes the update step through gradient descent on the loss function w.r.t. the patch.
  • Figure 3: The 10 optimized adversarial patches, along with their target labels.
  • Figure 4: A batch of clean images initially predicted correctly by a SqueezeNet iandola2016squeezenet model, and its perturbation with 2 different adversarial patches. Each row contains the original image with a different patch, whose target is displayed in the left. The predictions are shown on top of each of the samples, in green for correct prediction, blue for misclassification, and in red for a prediction that ends up in the target class of the attack.