On the Robustness of Vision Transformers to Adversarial Examples

TL;DR

This work assesses the adversarial robustness of Vision Transformers (ViTs) across white-box and black-box settings, comparing ViTs with Big Transfer (BiT) models and CNNs on CIFAR-10/100 and ImageNet. It shows that ViTs do not offer extra protection under white-box attacks, and that adversarial transferability between ViTs and non-transformers is limited. The authors introduce SAGA, a Self-Attention Gradient Attack, which defeats CNN–ViT ensembles in white-box scenarios, underscoring a security risk for simple ensembles. However, under black-box threat models, a ViT–BiT ensemble substantially improves robustness with minimal impact on clean accuracy, revealing a promising defense strategy leveraging low cross-genus transferability.

Abstract

Recent advances in attention-based networks have shown that Vision Transformers can achieve state-of-the-art or near state-of-the-art results on many image classification tasks. This puts transformers in the unique position of being a promising alternative to traditional convolutional neural networks (CNNs). While CNNs have been carefully studied with respect to adversarial attacks, the same cannot be said of Vision Transformers. In this paper, we study the robustness of Vision Transformers to adversarial examples. Our analyses of transformer security is divided into three parts. First, we test the transformer under standard white-box and black-box attacks. Second, we study the transferability of adversarial examples between CNNs and transformers. We show that adversarial examples do not readily transfer between CNNs and transformers. Based on this finding, we analyze the security of a simple ensemble defense of CNNs and transformers. By creating a new attack, the self-attention blended gradient attack, we show that such an ensemble is not secure under a white-box adversary. However, under a black-box adversary, we show that an ensemble can achieve unprecedented robustness without sacrificing clean accuracy. Our analysis for this work is done using six types of white-box attacks and two types of black-box attacks. Our study encompasses multiple Vision Transformers, Big Transfer Models and CNN architectures trained on CIFAR-10, CIFAR-100 and ImageNet.

Figures (9)

• Figure 1: Visual representation of Table \ref{['table:transfer']} for CIFAR-10. The x-axis corresponds to the model used to generate the adversarial examples. The y-axis corresponds to the model used to evaluate the adversarial examples. The z-axis measures transferability between the two models. The bars are color coded based on the two models. Pink, red, and orange bars represent the transferability between models of the same genus. Green, blue and light blue bars represent the transferability measurements between models of different genusus.
• Figure 2: Attack success rate of the Self-Attention Gradient Attack (SAGA), the Single MIM attack and Basic attack on an ensemble containing one ViT-L-16 model and one BiT-M-R101x3 model (or BiT-M-R152x4 for ImageNet). For full descriptions of each attack see Section \ref{['subsec:saga']}.
• Figure 3: Robust accuracy (higher is better) of different model configurations under black-box attacks. Here ViT/BiT is an ensemble containing a Vision Transformer (ViT-L-16) and a Big Transfer Model (BiT-M-101x3 for CIFAR-10/CIFAR-100 and Bit-M-R152x4 for ImageNet.
• Figure 4: Adversarial images generated using SAGA on CIFAR-10. The top row of images are the the clean images generated from the CIFAR-10 test set. The bottom row of images are the adversarial images generated using SAGA with the $l_{\infty}$ norm and $\epsilon=0.031$. These images correspond to SAGA when the models are ViT-L-16 and BiT-M-R101x3. Visually, there is very little perceivable difference between the clean and adversarial images generated by SAGA.
• Figure 5: Adversarial images generated using SAGA on ImageNet. The top row of images are the the clean images generated from the ImageNet validation set. The bottom row of images are the adversarial images generated using SAGA with the $l_{\infty}$ norm and $\epsilon=0.062$. These images correspond to SAGA when the models are ViT-L-16 and BiT-M-R152x4. Visually, there is very little perceivable difference between the clean and adversarial images generated by SAGA.