Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks

Yong Xie, Weijie Zheng, Hanxun Huang, Guangnan Ye, Xingjun Ma

TL;DR

This work tackles the challenge of robustly evaluating adversarial vulnerabilities at million-scale in image classification. It introduces Probability Margin Loss and the Probability Margin Attack (PMA), a stronger individual white-box attack, plus PMA-based ensembles to balance effectiveness and efficiency. Through extensive experiments on CIFAR-10/100 and ImageNet-1k with RobustBench models, PMA consistently outperforms existing losses and approaches AutoAttack in strength while being far faster. A million-scale evaluation on CC1M reveals significant robustness gaps between small-scale ImageNet testing and large-scale robustness, highlighting the practical importance of scalable robustness assessment for real-world deployments.

Abstract

As deep learning models are increasingly deployed in safety-critical applications, evaluating their vulnerabilities to adversarial perturbations is essential for ensuring their reliability and trustworthiness. Over the past decade, a large number of white-box adversarial robustness evaluation methods (i.e., attacks) have been proposed, ranging from single-step to multi-step methods and from individual to ensemble methods. Despite these advances, challenges remain in conducting meaningful and comprehensive robustness evaluations, particularly when it comes to large-scale testing and ensuring evaluations reflect real-world adversarial risks. In this work, we focus on image classification models and propose a novel individual attack method, Probability Margin Attack (PMA), which defines the adversarial margin in the probability space rather than the logits space. We analyze the relationship between PMA and existing cross-entropy or logits-margin-based attacks, and show that PMA can outperform the current state-of-the-art individual methods. Building on PMA, we propose two types of ensemble attacks that balance effectiveness and efficiency. Furthermore, we create a million-scale dataset, CC1M, derived from the existing CC3M dataset, and use it to conduct the first million-scale white-box adversarial robustness evaluation of adversarially-trained ImageNet models. Our findings provide valuable insights into the robustness gaps between individual versus ensemble attacks and small-scale versus million-scale evaluations.

Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks

TL;DR

This work tackles the challenge of robustly evaluating adversarial vulnerabilities at million-scale in image classification. It introduces Probability Margin Loss and the Probability Margin Attack (PMA), a stronger individual white-box attack, plus PMA-based ensembles to balance effectiveness and efficiency. Through extensive experiments on CIFAR-10/100 and ImageNet-1k with RobustBench models, PMA consistently outperforms existing losses and approaches AutoAttack in strength while being far faster. A million-scale evaluation on CC1M reveals significant robustness gaps between small-scale ImageNet testing and large-scale robustness, highlighting the practical importance of scalable robustness assessment for real-world deployments.

Abstract

As deep learning models are increasingly deployed in safety-critical applications, evaluating their vulnerabilities to adversarial perturbations is essential for ensuring their reliability and trustworthiness. Over the past decade, a large number of white-box adversarial robustness evaluation methods (i.e., attacks) have been proposed, ranging from single-step to multi-step methods and from individual to ensemble methods. Despite these advances, challenges remain in conducting meaningful and comprehensive robustness evaluations, particularly when it comes to large-scale testing and ensuring evaluations reflect real-world adversarial risks. In this work, we focus on image classification models and propose a novel individual attack method, Probability Margin Attack (PMA), which defines the adversarial margin in the probability space rather than the logits space. We analyze the relationship between PMA and existing cross-entropy or logits-margin-based attacks, and show that PMA can outperform the current state-of-the-art individual methods. Building on PMA, we propose two types of ensemble attacks that balance effectiveness and efficiency. Furthermore, we create a million-scale dataset, CC1M, derived from the existing CC3M dataset, and use it to conduct the first million-scale white-box adversarial robustness evaluation of adversarially-trained ImageNet models. Our findings provide valuable insights into the robustness gaps between individual versus ensemble attacks and small-scale versus million-scale evaluations.

Paper Structure

This paper contains 29 sections, 5 equations, 2 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. White-box Adversarial Attacks
  4. Reliable Adversarial Robustness Evaluation
  5. Proposed Attack
  6. Preliminary
  7. Probability Margin Loss
  8. Relationship to Existing Adversarial Losses
  9. Probability Margin Attack (PMA)
  10. Experiments
  11. Experimental Setup
  12. Datasets and Models
  13. Attack Setting and Baselines
  14. Main Results
  15. Effectiveness of the PM Loss
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Relative robustness evaluation on CC1M and ImageNet-1k test set. $x$-axis: top-5 (on RobustBench leaderboard) defense models adversarially trained on ImageNet; $y$-axis: relative robustness. Left: The red and blue lines represent the relative robustness measured on CC1M and ImageNet-1k test set, respectively. Right:The red and blue bars represent the relative robustness measured on CC1M by $\text{PGD}_{\text{ce}}$ and our PMA, respectively.
  • Figure 2: Visual illustrations of the attacked images.