Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement

TL;DR

The paper tackles the robustness gap of Vision Transformers to transferable adversarial examples by shifting surrogate refinement from backward gradients to forward propagation. It introduces Forward Propagation Refinement (FPR), comprising Attention Map Diversification (AMD) and Momentum Token Embedding (MTE), to modify attention maps and token embeddings during forward passes. Empirically, FPR outperforms state-of-the-art backward refinements by up to 7.0 percentage points on 13 ViT/CNN targets, remains effective under multiple defenses, and interoperates with other transfer-based attacks. The work provides both mechanistic insight into gradient behavior and a practical method to craft highly transferable adversarial examples, with available code for replication.

Abstract

Vision Transformers (ViTs) have been widely applied in various computer vision and vision-language tasks. To gain insights into their robustness in practical scenarios, transferable adversarial examples on ViTs have been extensively studied. A typical approach to improving adversarial transferability is by refining the surrogate model. However, existing work on ViTs has restricted their surrogate refinement to backward propagation. In this work, we instead focus on Forward Propagation Refinement (FPR) and specifically refine two key modules of ViTs: attention maps and token embeddings. For attention maps, we propose Attention Map Diversification (AMD), which diversifies certain attention maps and also implicitly imposes beneficial gradient vanishing during backward propagation. For token embeddings, we propose Momentum Token Embedding (MTE), which accumulates historical token embeddings to stabilize the forward updates in both the Attention and MLP blocks. We conduct extensive experiments with adversarial examples transferred from ViTs to various CNNs and ViTs, demonstrating that our FPR outperforms the current best (backward) surrogate refinement by up to 7.0\% on average. We also validate its superiority against popular defenses and its compatibility with other transfer methods. Codes and appendix are available at https://github.com/RYC-98/FPR.

Figures (5)

• Figure 1: Our Forward Propagation Refinement (FPR) vs. previous methods which only refine the backward propagation.
• Figure 2: The overview of Forward Propagation Refinement (FPR). Note that Attention Map Diversification (AMD) is applied to specific attention maps according to the index set, while Momentum Token Embedding (MTE) is applied to all Attention and MLP blocks.
• Figure 3: Illustration of redundancy in attention maps: Randomly dropping a small number of blocks leads to improved transferability. The results are averaged over 5 repeated runs on 13 target models, and the dash lines denote the baseline results of the MIM mi without dropping.
• Figure 4: Sensitivity analysis about diversity factor, scale factor, and attenuation factor, where ASR remains relatively stable within a certain range of values for these three factors.
• Figure 5: Visualizations for the crafted adversarial examples. Here the substitute model is ViT-B and the target model is RN-18.