Transferable Adversarial Face Attack with Text Controlled Attribute

TL;DR

This work targets unrestricted, text-guided impersonation attacks on face recognition by integrating CLIP-based text guidance with StyleGAN2 latent manipulation. The method, TCA^2, uses a category-level softmax vector to steer impersonation toward a target identity while preserving perceptual similarity, and it leverages data and model augmentation, including a meta-learning framework, to boost transferability to unseen FR models. Extensive experiments on CelebA-Identity and KID-F demonstrate high impersonation ASR and competitive image quality, with successful transfer to commercial APIs such as Face++ and Aliyun. The results highlight both the practical feasibility of text-controlled adversarial faces and the need for defense strategies against unrestricted, semantically guided attacks in real-world systems.

Abstract

Traditional adversarial attacks typically produce adversarial examples under norm-constrained conditions, whereas unrestricted adversarial examples are free-form with semantically meaningful perturbations. Current unrestricted adversarial impersonation attacks exhibit limited control over adversarial face attributes and often suffer from low transferability. In this paper, we propose a novel Text Controlled Attribute Attack (TCA$^2$) to generate photorealistic adversarial impersonation faces guided by natural language. Specifically, the category-level personal softmax vector is employed to precisely guide the impersonation attacks. Additionally, we propose both data and model augmentation strategies to achieve transferable attacks on unknown target models. Finally, a generative model, \textit{i.e}, Style-GAN, is utilized to synthesize impersonated faces with desired attributes. Extensive experiments on two high-resolution face recognition datasets validate that our TCA$^2$ method can generate natural text-guided adversarial impersonation faces with high transferability. We also evaluate our method on real-world face recognition systems, \textit{i.e}, Face++ and Aliyun, further demonstrating the practical potential of our approach.

Figures (19)

• Figure 1: The visualization of source face, other different adversarial face and target face. The second image is from GAN inversion. Some representations of norm-based adversarial examplesmadry2017towards and unrestricted adversarial exampleskomkov2021advhat are shown. TCA$^2$ generates the 5$^{th}$ image.
• Figure 2: The overall framework of our proposed Text Controlled Attribute Attack (TCA$^2$). Our proposed method involves searching for adversarial faces on the generative StyleGAN manifold by optimizing the parameters of a Fusion Network. The generated photo-realistic adversarial faces can deceive state-of-the-art face recognition (FR) systems under the guidance of text prompts. To effectively represent the semantics of an impersonated person, a softmax vector is employed to perform a targeted attack against the FR system. The FR models in the framework are randomly selected from either the meta-train set or the meta-test set.
• Figure 3: The visualization of source face, other different adversarial face and target face. Each image row is source, MI-FGSM, PGD, AdvHat, TCA$^2$, and target face, respectively.
• Figure 4: Mean confidence scores returned from commercial APIs, i.e, Face++ and Aliyun.
• Figure 5: The visualizations of the text prompt's style on the visual quality of the output images. These images successfully deceive the facial recognition (FR) model. The text prompt used was "A female face with open mouth".