Boosting the Local Invariance for Better Adversarial Transferability

TL;DR

The paper tackles transfer-based adversarial attacks by revealing that adversarial perturbations exhibit weaker translation invariance than clean images, and that higher local invariance correlates with better cross-model transferability. It introduces Local Invariance Boosting (LI-Boost), which optimizes perturbations under $||\delta||_p \leq \epsilon$ to maximize the minimum loss over a $k\times k$ neighborhood of translations, with gradients approximated via random sampling $\bar{g} = \frac{1}{N} \sum \nabla_{x+\delta} J(x+\Gamma(\delta,i,j), y; \theta)$. Through extensive ImageNet experiments, LI-Boost consistently boosts transferability of gradient-based, input transformation-based, model-related, advanced Objective, and ensemble attacks across CNNs and ViTs, even against defenses. The approach provides a general, practical direction for enhancing adversarial transferability and highlights a new attacker-centric lens on perturbation–image coupling for robustness analysis.

Abstract

Transfer-based attacks pose a significant threat to real-world applications by directly targeting victim models with adversarial examples generated on surrogate models. While numerous approaches have been proposed to enhance adversarial transferability, existing works often overlook the intrinsic relationship between adversarial perturbations and input images. In this work, we find that adversarial perturbation often exhibits poor translation invariance for a given clean image and model, which is attributed to local invariance. Through empirical analysis, we demonstrate that there is a positive correlation between the local invariance of adversarial perturbations w.r.t. the input image and their transferability across different models. Based on this finding, we propose a general adversarial transferability boosting technique called Local Invariance Boosting approach (LI-Boost). Extensive experiments on the standard ImageNet dataset demonstrate that LI-Boost could significantly boost various types of transfer-based attacks (e.g., gradient-based, input transformation-based, model-related, advanced objective function, ensemble, etc.) on CNNs, ViTs, and defense mechanisms. Our approach presents a promising direction for future research in improving adversarial transferability across different models.

Figures (4)

• Figure 1: The impact of translation invariance of clean image and adversarial perturbation. The translated clean image can be correctly recognized by deep models (either CNNs or ViTs) while the translated adversarial perturbations cannot consistently fool the deep models.
• Figure 2: The local invariance of various transfer-based attacks (left) and the attack performance of MI-FGSM with enhanced local invariance (right).
• Figure 3: Ablation studies of various sampling distributions.
• Figure 4: Ablation studies of number of sampled perturbations $N$ and upper bound of translated pixels $k$.

Theorems & Definitions (1)

• Definition 1: Local Invariance