Stop Walking in Circles! Bailing Out Early in Projected Gradient Descent

TL;DR

The paper addresses the high computational cost of evaluating adversarial robustness with PGD under the $L_ty$ ball. It introduces PGD$_{CD}$, a simple cycle-detection based early-termination method that halts PGD when a perturbation repeats, preserving the exact robustness estimate while substantially reducing iterations. Empirical results across RobustBench models on ImageNet and CIFAR datasets show 10x–20x speedups with robustness estimates matching standard PGD, enabling scalable robustness evaluations and faster experimentation. The approach remains competitive against stronger attacks like Auto-PGD and facilitates acceleration in related tasks such as adversarial training, reducing compute and energy requirements.

Abstract

Projected Gradient Descent (PGD) under the $L_\infty$ ball has become one of the defacto methods used in adversarial robustness evaluation for computer vision (CV) due to its reliability and efficacy, making a strong and easy-to-implement iterative baseline. However, PGD is computationally demanding to apply, especially when using thousands of iterations is the current best-practice recommendation to generate an adversarial example for a single image. In this work, we introduce a simple novel method for early termination of PGD based on cycle detection by exploiting the geometry of how PGD is implemented in practice and show that it can produce large speedup factors while providing the \emph{exact} same estimate of model robustness as standard PGD. This method substantially speeds up PGD without sacrificing any attack strength, enabling evaluations of robustness that were previously computationally intractable.

Figures (5)

• Figure 1: The key insight of our work is that, under the constraint $\|\delta_t\|_\infty < \epsilon$, when it is not possible to find an adversarial example such that $f(x+\delta_t) \neq f(x)$, it is often the case that the PGD optimization will begin to cycle. That is for a given iterate $t$, and an offset $o$, we find $\delta_t = \delta_{t+o}$. We can save significant computational resources by detecting the cycles and stopping the PGD attack early. The perturbation $\delta$ in this figure is exaggerated for visibility.
• Figure 2: A 2D example of how a cycle of length two can occur on the boundary of the $L_\infty$ ball. The minimum-norm adversarial example $\boldsymbol{x}^*$ is outside the $\epsilon$ ball, so the original datum $\boldsymbol{x}$ is robust. The gradients point toward the optimal solution so that the projected signed gradient steps oscillate between the same two points on the boundary, one on each side of the optimum.
• Figure 3: Cosine similarities ($S_C$) between signed gradients using the Carmon2019Unlabeledcarmon2019unlabeled CIFAR10 model from RobustBench. The $S_C$ between the first and second preceding items (lag=1, 2) all converge to a constant when a cycle forms. When the lag is a multiple of the cycle length, we will, by definition, reach a $S_C$ of 1 as it compares to previous iterations of itself.
• Figure 4: Two-dimensional UMAP projection of perturbations $\delta^{(i)}$ generated from different PGD iterations for the Carmon2019Unlabeled CIFAR10 model from RobustBench. Top: 1000 PGD iterations are shown, with a length-two cycle occurring after 845 iterations and alternating between the red and blue points. Bottom: 1000 PGD iterations are shown, and a length-four cycle occurs after 934 iterations and cycles across the green, cyan, blue, and red points (in that order). Black points correspond to perturbations before the cycle begins, and colored points correspond to the perturbations between which PGD ultimately oscillates.
• Figure 5: Plots of the percentage of iterations that cycle detection reduces in PGD against the maximum iteration budget per image, $T_{\rm iter}$, for CIFAR10, CIFAR100, and ImageNet. Cycle detection obviously results in larger computational savings for larger iteration budgets, but most models enjoy significant computational savings even for reasonably small budgets like $T_{\rm iter}=100.$