Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume

TL;DR

This work proposes a new metric termed as the adversarial hypervolume for assessing the robustness of deep learning models comprehensively over a range of perturbation intensities from a multi-objective optimization standpoint and adopts a novel training algorithm to enhance adversarial robustness uniformly across various perturbation intensities.

Abstract

The escalating threat of adversarial attacks on deep learning models, particularly in security-critical fields, has underscored the need for robust deep learning systems. Conventional robustness evaluations have relied on adversarial accuracy, which measures a model's performance under a specific perturbation intensity. However, this singular metric does not fully encapsulate the overall resilience of a model against varying degrees of perturbation. To address this gap, we propose a new metric termed adversarial hypervolume, assessing the robustness of deep learning models comprehensively over a range of perturbation intensities from a multi-objective optimization standpoint. This metric allows for an in-depth comparison of defense mechanisms and recognizes the trivial improvements in robustness afforded by less potent defensive strategies. Additionally, we adopt a novel training algorithm that enhances adversarial robustness uniformly across various perturbation intensities, in contrast to methods narrowly focused on optimizing adversarial accuracy. Our extensive empirical studies validate the effectiveness of the adversarial hypervolume metric, demonstrating its ability to reveal subtle differences in robustness that adversarial accuracy overlooks. This research contributes a new measure of robustness and establishes a standard for assessing and benchmarking the resilience of current and future defensive models against adversarial threats.

Figures (7)

• Figure 1: Comparison between adversarial hypervolume and established robustness metrics.Adversarial sparsity measures the proportion of non-adversarial to total examples at perturbation level $\epsilon$. Probability accuracy denotes the ratio of high-confidence predictions to total examples at the perturbation level $\epsilon$. Adversarial hypervolume represents the averaged variations in confidence scores over a range of perturbation intensities.
• Figure 2: An example of adversarial frontiers. The adversarial frontiers of different models computed using a randomly selected image from CIFAR-10 test set. R is for ResNet and WR is for WideResNet. The label represents the name of the model in the RobustBench library.
• Figure 3: An illustration of the adversarial hypervolume. Intuitively, it functions as an approximation of the integral of confidence values, where the gray area represents the approximation's residual error.
• Figure 4: Accuracy Comparison. Results of the clean and robust accuracy trained under fixed budget and ascending budget strategy.
• Figure 5: An illustration of computed and theoretical error between the adversarial hypervolume and the true integral of the confidence loss (Adversarial hypervolume approximated using $N=20$ points as a proxy). The computed values are represented by the blue line, while the theoretical error function is depicted with a red dashed line.