On the Adversarial Robustness of Vision Transformers

TL;DR

This work provides a comprehensive investigation into the adversarial robustness of Vision Transformers (ViTs) relative to CNNs and MLP-Mixer, using frequency-domain analyses and denoised randomized smoothing to explain and certify robustness. Key findings show ViTs rely less on high-frequency features, which helps resist high-frequency adversarial perturbations, and that introducing CNN/T2T blocks or increasing transformer proportions in hybrids can trade robustness for higher clean accuracy. The study also demonstrates that adversarial training is applicable to ViTs and that SAM further enhances robustness, while basic pretraining on larger datasets does not substantially improve robustness. Together, the results offer design guidance for robust vision models, indicating that modern CNNs inspired by ViT principles can bridge the robustness gap, and that frequency-focused analyses are valuable for understanding and improving model resilience.

Abstract

Following the success in advancing natural language processing and understanding, transformers are expected to bring revolutionary changes to computer vision. This work provides a comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with MLP-Mixer and convolutional neural networks (CNNs) including ConvNeXt, and this observation also holds for certified robustness. Through frequency analysis and feature visualization, we summarize the following main observations contributing to the improved robustness of ViTs: 1) Features learned by ViTs contain less high-frequency patterns that have spurious correlation, which helps explain why ViTs are less sensitive to high-frequency perturbations than CNNs and MLP-Mixer, and there is a high correlation between how much the model learns high-frequency features and its robustness against different frequency-based perturbations. 2) Introducing convolutional or tokens-to-token blocks for learning high-frequency features in ViTs can improve classification accuracy but at the cost of adversarial robustness. 3) Modern CNN designs that borrow techniques from ViTs including activation function, layer norm, larger kernel size to imitate the global attention, and patchify the images as inputs, etc., could help bridge the performance gap between ViTs and CNNs not only in terms of performance, but also certified and empirical adversarial robustness. Moreover, we show adversarial training is also applicable to ViT for training robust models, and sharpness-aware minimization can also help improve robustness, while pre-training with clean images on larger datasets does not significantly improve adversarial robustness.

Figures (8)

• Figure 1: Robust accuracy v.s. clean accuracy. The robust accuracy is evaluated by AutoAttack auto-attack. The "Hybrid" class includes CNN-ViT, T2T-ViT and Swin-T as introduced in Section \ref{['sec:model']}. Models with attention mechanisms have their names printed at the center of the circles. ViTs have the best robustness against adversarial perturbations. Introducing other modules to ViT can improve clean accuracy but hurt adversarial robustness. CNNs are more vulnerable to adversarial attacks.
• Figure 2: Filters for the frequency-based attack. The frequencies corresponding to the red part are filtered out, and the frequencies corresponding to the green part can pass through. "Full Pass" means all of the frequencies are preserved. "Low Pass" means only low-frequency components are preserved. "High Pass" preserves the high-frequency part.
• Figure 3: Feature visualization: The learned low-level structure features are highlighted in blue (obviously perceptible) and green (minorly perceptible). The CNNs in the first row learn more low-level features compared with the ViTs in the second row. The ViTs pay more attention to the low-level structures and their feature maps become noisier when ResNet features are introduced (ViT-B/16-Res) or neighboring tokens are aggregated into one token recursively (T2T-ViT-24).
• Figure 4: Target model error rate on adversarial examples (i.e., 1.0 - RA) against transfer attack using FGSM with different attack radii. The rows stand for the surrogate models used to generated adversarial examples. The columns stand for the target models. Darker rows correlate to the source models that generate more transferable adversarial examples. Darker columns mean that the target models are more vulnerable to transfer attack."Res50-ssl" and "Res50-swsl" are in short of "ResNeXt-32x4d-ssl" and "ResNet50-swsl" respectively. Results for more radii can be found in Appendix \ref{['app:transfer']}.
• Figure 5: Cross entropy loss versus varying PGD attack steps for ViT-S/16 and RestNet18. The dashed lines corresponds to larger attach radius of $0.03$ and the full lines to smaller attack radius of $0.01$.