Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

TL;DR

The paper tackles the problem of adversarial vulnerability in Vision Transformers (ViTs) under hard-label black-box settings. It introduces AdvViT, a query-efficient attack that targets patch-level perturbations by operating on low-frequency components of each patch through a block DCT transform and a variance-based weight mask to guide perturbations, achieving lower $L_2$ distortion than CNN-based attacks under the same query budget. The approach combines a three-step low-frequency optimization in the DCT domain with a Sign-OPT-based baseline (AdvViT uses Sign-OPT, AdvViT+ uses Sign-OPT+), and demonstrates effectiveness across multiple ViT backbones on ImageNet-1K, with improvements in PSNR/SSIM indicating high perceptual stealth. These results challenge prior claims of ViT robustness and provide a ViT-specific adversarial framework that informs both attack strategies and robustness evaluations in transformer-based vision models.

Abstract

Recent studies have revealed that vision transformers (ViTs) face similar security risks from adversarial attacks as deep convolutional neural networks (CNNs). However, directly applying attack methodology on CNNs to ViTs has been demonstrated to be ineffective since the ViTs typically work on patch-wise encoding. This article explores the vulnerability of ViTs against adversarial attacks under a black-box scenario, and proposes a novel query-efficient hard-label adversarial attack method called AdvViT. Specifically, considering that ViTs are highly sensitive to patch modification, we propose to optimize the adversarial perturbation on the individual patches. To reduce the dimension of perturbation search space, we modify only a handful of low-frequency components of each patch. Moreover, we design a weight mask matrix for all patches to further optimize the perturbation on different regions of a whole image. We test six mainstream ViT backbones on the ImageNet-1k dataset. Experimental results show that compared with the state-of-the-art attacks on CNNs, our AdvViT achieves much lower $L_2$-norm distortion under the same query budget, sufficiently validating the vulnerability of ViTs against adversarial attacks.

Figures (6)

• Figure 1: Overview of the proposed AdvViT method.
• Figure 2: Attack in Low-frequency DCT Domain
• Figure 3: Query-Efficient Hard-label Black-box Attack against Vision Transformers
• Figure 4: Results of AdvViT+ with different $\rho$ values on the Deit-T model. Within the maximum number of model queries 4000: (a) is the number of samples that can successfully find adversarial examples in 100 tested images; (b) shows the success rates on the given thresholds ($\epsilon$) 3.0, 5.0 and 8.0, respectively; (c) gives out the $L_2$ loss curves, including average $L_2$ loss and median $L_2$ loss.
• Figure 5: Results of AdvViT and AdvViT+ with different $\alpha$ values on the Deit-T model. The maximum number of model queries is 4000 and the dimension reduction ratio $\rho=3/16$. (a) and (b) show the $L_2$ loss curves of AdvViT and AdvViT+, respectively. (c) shows the success rate curves on the given threshold $\epsilon =5.0$.