Reveal of Vision Transformers Robustness against Adversarial Attacks

TL;DR

This work systematically evaluates the adversarial robustness of Vision Transformer (ViT) variants against a broad set of attacks, including white-box, black-box, gray-box, and EOT scenarios, and compares them with CNN baselines. By combining attack experiments with preprocessing defenses and analysis tools like DCT perturbation spectra and attention/GCAM visualizations, the study shows vanilla ViT and hybrid-ViT generally exhibit greater resilience than CNNs under many $L_p$-norm attacks. Key findings include the superior robustness of Vanilla ViT to $L_0$ and cw-$L_2$ attacks in certain configurations, the strong performance of t2t-24 and tnt-S-16 against $L_1$ attacks, and the nuanced impact of preprocessing defenses and tokenization on robustness. The results highlight that attack transferability is mitigated by deeper attention blocks, while preprocessing defenses and CCP offer mixed benefits depending on model architecture, with significant practical implications for deploying ViT-based systems in adversarial settings.

Abstract

The major part of the vanilla vision transformer (ViT) is the attention block that brings the power of mimicking the global context of the input image. For better performance, ViT needs large-scale training data. To overcome this data hunger limitation, many ViT-based networks, or hybrid-ViT, have been proposed to include local context during the training. The robustness of ViTs and its variants against adversarial attacks has not been widely investigated in the literature like CNNs. This work studies the robustness of ViT variants 1) against different Lp-based adversarial attacks in comparison with CNNs, 2) under adversarial examples (AEs) after applying preprocessing defense methods and 3) under the adaptive attacks using expectation over transformation (EOT) framework. To that end, we run a set of experiments on 1000 images from ImageNet-1k and then provide an analysis that reveals that vanilla ViT or hybrid-ViT are more robust than CNNs. For instance, we found that 1) Vanilla ViTs or hybrid-ViTs are more robust than CNNs under Lp-based attacks and under adaptive attacks. 2) Unlike hybrid-ViTs, Vanilla ViTs are not responding to preprocessing defenses that mainly reduce the high frequency components. Furthermore, feature maps, attention maps, and Grad-CAM visualization jointly with image quality measures, and perturbations' energy spectrum are provided for an insight understanding of attention-based models.

Figures (22)

• Figure 1: The asr of target models, on 1000 images from ImageNet-1k, against AutoAttack, in average $\epsilon=\{1,2,4\}/255$, and the target model top-1 error of the preprocessed ae for six different preprocessing defense methods including SS: ss, NLM: nlm, TVM: tvm, JPEG: jpg, CR: cr and CCP: ccp.
• Figure 2: The perturbation (top), generated using cw-$L_\infty$ attack with vit-B-16 (middle) and ResNet50(left), and the corresponding dct-based spectral decomposition heatmap. Perturbation is scaled from [-1, 1] to [0, 255].
• Figure 3: The perturbation (top), generated using pgd-$L_\infty$$\epsilon=4/255$ attack with t2t-14 (middle) and ResNet50(left), and the corresponding dct-based spectral decomposition heatmap. Perturbation is scaled from [-1, 1] to [0, 255].
• Figure 4: The asr of target models, on 1000 images from ImageNet-1k against AutoAttack, in average $\epsilon=\{1,2,4\}/255$, and the mad
• Figure 5: jsm attack: (a) ae quality assessment measures. (b) The asr of the ae and the top-1 error of the pre-processed ae on 100 images from imagenet-1k. SS: ss. NLM: nlm. TVM: tvm. JPEG: jpg. CR: cr. CCP: ccp.