Scaling Adversarial Training via Data Selection

TL;DR

Adversarial training with PGD is robust but computationally heavy since all samples are attacked. The authors propose selective adversarial training, using margin-based and gradient-matching criteria to perturb only a subset of samples and a mixed loss to keep clean accuracy. On MNIST and CIFAR-10, both strategies achieve robustness close to full PGD while reducing adversarial computation by up to roughly 2x–4x depending on dataset. This work demonstrates that principled data selection can scale adversarial robustness without sacrificing much performance.

Abstract

Projected Gradient Descent (PGD) is a strong and widely used first-order adversarial attack, yet its computational cost scales poorly, as all training samples undergo identical iterative inner-loop optimization despite contributing unequally to robustness. Motivated by this inefficiency, we propose \emph{Selective Adversarial Training}, which perturbs only a subset of critical samples in each minibatch. Specifically, we introduce two principled selection criteria: (1) margin-based sampling, which prioritizes samples near the decision boundary, and (2) gradient-matching sampling, which selects samples whose gradients align with the dominant batch optimization direction. Adversarial examples are generated only for the selected subset, while the remaining samples are trained cleanly using a mixed objective. Experiments on MNIST and CIFAR-10 show that the proposed methods achieve robustness comparable to, or even exceeding, full PGD adversarial training, while reducing adversarial computation by up to $50\%$, demonstrating that informed sample selection is sufficient for scalable adversarial robustness.