Attention-aggregated Attack for Boosting the Transferability of Facial Adversarial Examples

TL;DR

This work tackles the transferability limitation of adversarial perturbations against face recognition (FR) in black-box settings. It introduces Attention-aggregated Attack (AAA), a two-stage, feature-level attack that first transfers attentions from a source FR model to others via iterative perturbations and then aggregates these transferred attentions across $N$ steps to cover multiple facial features. The authors show that decisive and auxiliary facial features are model-specific, and formalize AAA with an attention-based loss $L(x^{adv})=\sum\left(\sum_{k=0}^{N-1}\frac{\partial \mathcal{D}_{f_\theta}(x^{adv}_k,x^{r})}{h^{adv}_k}\right)\odot h^{adv}$, trained under a budget $\norm{x^{adv}-x^{i}}_\infty<\epsilon$ using momentum-based updates. Extensive experiments across diverse FR backbones, boosting techniques (DI-SIM, SG, etc.), and model ensembles demonstrate that AAA substantially improves transfer-based attack performance over baselines like MIM and LGC, with consistent gains in both dodging and impersonation, and clear guidance on layer selection and perturbation budgets for practical deployment.

Abstract

Adversarial examples have revealed the vulnerability of deep learning models and raised serious concerns about information security. The transfer-based attack is a hot topic in black-box attacks that are practical to real-world scenarios where the training datasets, parameters, and structure of the target model are unknown to the attacker. However, few methods consider the particularity of class-specific deep models for fine-grained vision tasks, such as face recognition (FR), giving rise to unsatisfactory attacking performance. In this work, we first investigate what in a face exactly contributes to the embedding learning of FR models and find that both decisive and auxiliary facial features are specific to each FR model, which is quite different from the biological mechanism of human visual system. Accordingly we then propose a novel attack method named Attention-aggregated Attack (AAA) to enhance the transferability of adversarial examples against FR, which is inspired by the attention divergence and aims to destroy the facial features that are critical for the decision-making of other FR models by imitating their attentions on the clean face images. Extensive experiments conducted on various FR models validate the superiority and robust effectiveness of the proposed method over existing methods.

Figures (3)

• Figure 1: Visualization of the up-sampled gradients (visual importance) of the loss function (cosine similarity) w.r.t. the feature maps in certain mid-layers, which are overlying on the input face image. We select layer2.3 in ResNet50, layer.7 in MobileNet. It can be found that different FRs concentrate on distinct preferable facial features. Gradients of $x^{adv}_k$ displayed in the second and third rows indicate that the attention of a specific model keeps transferring with iteratively added adversarial noise. Considering that the facial features are orthogonal to each other in the pixel space, averaging these diverted visual importance achieves a special kind of ensemble-like effect.
• Figure 2: The influence of layer choice on ASR. Shallow, middle, and deep layers are respectively attacked in Softmax-IR to craft adversarial examples with our method whose success rates against different target models are exhibited.
• Figure 3: The influence of the budget of adversarial noise on attacking performance. ASRs against nine target models are exhibited for selecting a proper budget of adversarial noise.