Towards Robust Vision Transformer via Masked Adaptive Ensemble

TL;DR

The paper tackles the vulnerability of Vision Transformers to adversarial attacks by proposing a detector–classifier ViT architecture connected via an adaptive ensemble. The detector uses a Guided Backpropagation–based Multi-head Self-Attention and a Soft-Nearest Neighbors loss to expose adversarial perturbations, while the classifier employs two encoders (clean and adversarial) with MAE-inspired pre-training and a decoder, followed by fine-tuning with an adaptive patch-masking ensemble. A two-stage training regime—pre-training and fine-tuning—along with a novel adaptive ensemble bridges detector outputs to the classifier, enabling robust performance under adaptive attacks with minimal degradation in standard accuracy. Experimental results on CIFAR-10, CIFAR-100, and Tiny-ImageNet show state-of-the-art standard accuracy and adversarial robustness, validating the approach’s effectiveness across diverse datasets and attack settings.

Abstract

Adversarial training (AT) can help improve the robustness of Vision Transformers (ViT) against adversarial attacks by intentionally injecting adversarial examples into the training data. However, this way of adversarial injection inevitably incurs standard accuracy degradation to some extent, thereby calling for a trade-off between standard accuracy and robustness. Besides, the prominent AT solutions are still vulnerable to adaptive attacks. To tackle such shortcomings, this paper proposes a novel ViT architecture, including a detector and a classifier bridged by our newly developed adaptive ensemble. Specifically, we empirically discover that detecting adversarial examples can benefit from the Guided Backpropagation technique. Driven by this discovery, a novel Multi-head Self-Attention (MSA) mechanism is introduced to enhance our detector to sniff adversarial examples. Then, a classifier with two encoders is employed for extracting visual representations respectively from clean images and adversarial examples, with our adaptive ensemble to adaptively adjust the proportion of visual representations from the two encoders for accurate classification. This design enables our ViT architecture to achieve a better trade-off between standard accuracy and robustness. Besides, our adaptive ensemble technique allows us to mask off a random subset of image patches within input data, boosting our ViT's robustness against adaptive attacks, while maintaining high standard accuracy. Experimental results exhibit that our ViT architecture, on CIFAR-10, achieves the best standard accuracy and adversarial robustness of 90.3% and 49.8%, respectively.

Figures (7)

• Figure 1: Visualizations on the clean image (Top) and the adversarial example (Bottom). Left: Original clean image and adversarial example. Middle: Guided Grad-CAM visualization. Right: Guided Backpropagation visualization.
• Figure 2: Our model architecture: (a) detector and (b) classifier during the pre-training stage.
• Figure 3: Illustration of our model architecture during the fine-tuning stage.
• Figure 4: Performance stability under different datasets and different adaptive attacks.
• Figure 5: Comparison of the reconstruction quality from masked adversarial examples by employing our approach with/without the contrastive loss, denoted as our approach (w/ CL) and our approach (w/o CL), respectively. From left to right are the original adversarial example, the masked input, reconstruction by our approach (w/o CL), and reconstruction by our approach (w/ CL), respectively.