Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features

TL;DR

The paper tackles the problem of black-box adversarial transferability by arguing that perturbations should disrupt shared latent representations rather than labels. It introduces dSVA, a generative attack that leverages dual self-supervised Vision Transformer features from DINO (CL) and MAE (MIM) through facet-level exploitation, self-attention regularization, and joint feature discrimination. Empirical results show that dSVA achieves state-of-the-art black-box transferability across diverse architectures, including ConvNets, ViTs, and MLP-Mixers, and remains effective against defended models. These findings highlight the robustness implications of self-supervised ViTs and suggest that exploiting their dual representations can yield highly transferable adversarial perturbations across a wide range of models and defenses.

Abstract

The ability of deep neural networks (DNNs) come from extracting and interpreting features from the data provided. By exploiting intermediate features in DNNs instead of relying on hard labels, we craft adversarial perturbation that generalize more effectively, boosting black-box transferability. These features ubiquitously come from supervised learning in previous work. Inspired by the exceptional synergy between self-supervised learning and the Transformer architecture, this paper explores whether exploiting self-supervised Vision Transformer (ViT) representations can improve adversarial transferability. We present dSVA -- a generative dual self-supervised ViT features attack, that exploits both global structural features from contrastive learning (CL) and local textural features from masked image modeling (MIM), the self-supervised learning paradigm duo for ViTs. We design a novel generative training framework that incorporates a generator to create black-box adversarial examples, and strategies to train the generator by exploiting joint features and the attention mechanism of self-supervised ViTs. Our findings show that CL and MIM enable ViTs to attend to distinct feature tendencies, which, when exploited in tandem, boast great adversarial generalizability. By disrupting dual deep features distilled by self-supervised ViTs, we are rewarded with remarkable black-box transferability to models of various architectures that outperform state-of-the-arts. Code available at https://github.com/spencerwooo/dSVA.

Figures (9)

• Figure 1: Demonstration of dSVA. By jointly exploiting deep features of both self-supervised ViTs, i.e., DINO (CL) and MAE (MIM), dSVA crafts perturbation that disrupts both structural and textural representations of the image (as visualized in the attention saliency maps), fooling ConvNets, ViTs, and MLPs alike.
• Figure 2: The dSVA Training Framework. Sample $\bm{x}$ is fed through $\mathcal{G}$ to create adversarial example $\bm{x}^{adv}$, which are then both fed into the self-supervised models DINO and MAE, to extract deep representations and attention saliency maps from both global structural and local textural feature aspects. The feature discriminative loss is derived from both ViTs, which jointly form the adversarial loss $\mathcal{L}_\text{adv}$.
• Figure 3: Attention saliency maps. We visualize the attention saliency maps derived from both self-supervised ViTs DINO (first row) and MAE (second row), and the supervised ViT (third row). From left to right, layer depth increase from shallow to deep (from 1 to 11).
• Figure 4: Impact of the choice of facet. We evaluate the transferability of dSVA (DINO) and dSVA (MAE) that exploit feature facets at layer 10 of query, key, value, and token, respectively.
• Figure 5: Impact of the choice of layer $l$. We evaluate the transferability of dSVA (DINO) and dSVA (MAE) with layer $l$ from 1 to 11 (from left to right).