Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability

TL;DR

The paper investigates why adversarial perturbations crafted on Vision Transformers (ViTs) transfer well and proposes exploiting ViT computational redundancy to boost transferability. It identifies two forms of redundancy—data-level (token-level) and model-level (over-parameterization, dropout, head diversity)—and develops a suite of redundancy-driven attack techniques, including attention sparsity, head permutation, clean token regularization, ghost MoE, and test-time adversarial training, complemented by offline robust tokens. An online learning strategy based on policy gradients selects and optimizes these operations across transformer blocks to maximize adversarial loss, achieving strong cross-model transferability on ImageNet-1k across ViTs and CNNs. The results reveal substantial improvements over baselines and demonstrate the generality of the approach across architectures, highlighting a new axis for adversarial vulnerability in ViTs and informing defense design against transferable attacks.

Abstract

Vision Transformers (ViTs) have demonstrated impressive performance across a range of applications, including many safety-critical tasks. However, their unique architectural properties raise new challenges and opportunities in adversarial robustness. In particular, we observe that adversarial examples crafted on ViTs exhibit higher transferability compared to those crafted on CNNs, suggesting that ViTs contain structural characteristics favorable for transferable attacks. In this work, we investigate the role of computational redundancy in ViTs and its impact on adversarial transferability. Unlike prior studies that aim to reduce computation for efficiency, we propose to exploit this redundancy to improve the quality and transferability of adversarial examples. Through a detailed analysis, we identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness. Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and test-time adversarial training. Extensive experiments on the ImageNet-1k dataset validate the effectiveness of our approach, showing that our methods significantly outperform existing baselines in both transferability and generality across diverse model architectures.

Figures (7)

• Figure 1: Overview of the proposed attack strategy integrated into Vision Transformers (ViTs). Our method adopts a policy gradient-based framework to selectively apply different operations from an operation pool to each transformer block. These operations include permuting attention heads, sparsifying them, clean token regularization, and activating auxiliary Ghost MoE branches to exploit the computational redundancy within ViTs. Robust tokens are learned at test time to further enhance adversarial transferability.
• Figure 2: Study on the effectiveness of randomly dropping attention weights.
• Figure 3: Study on the effectiveness of shuffling attention heads.
• Figure 4: Study on the effectiveness of clean tokens in regularization.
• Figure 5: Study on the effectiveness of diversifying the FFN.