Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks

TL;DR

Adv-CPG tackles the privacy risks of customized portrait generation by merging facial adversarial attacks with text-guided synthesis. It introduces a three-component framework—ID Encryptor En1, Encryption Enhancer En2, and Multi-Modal Image Customizer MMIC—and a two-stage generation pipeline that first protects privacy and then enables fine-grained portrait customization. Training focuses on MMIC and En1, while inference uses a delayed conditioning scheme to balance identity protection with semantic control. Empirical results show strong black-box attack performance against multiple FR models and commercial APIs, while still delivering high-fidelity, personalized portraits, highlighting a practical approach to safeguarding facial privacy in AI-enabled portrait tools.

Abstract

Recent Customized Portrait Generation (CPG) methods, taking a facial image and a textual prompt as inputs, have attracted substantial attention. Although these methods generate high-fidelity portraits, they fail to prevent the generated portraits from being tracked and misused by malicious face recognition systems. To address this, this paper proposes a Customized Portrait Generation framework with facial Adversarial attacks (Adv-CPG). Specifically, to achieve facial privacy protection, we devise a lightweight local ID encryptor and an encryption enhancer. They implement progressive double-layer encryption protection by directly injecting the target identity and adding additional identity guidance, respectively. Furthermore, to accomplish fine-grained and personalized portrait generation, we develop a multi-modal image customizer capable of generating controlled fine-grained facial features. To the best of our knowledge, Adv-CPG is the first study that introduces facial adversarial attacks into CPG. Extensive experiments demonstrate the superiority of Adv-CPG, e.g., the average attack success rate of the proposed Adv-CPG is 28.1% and 2.86% higher compared to the SOTA noise-based attack methods and unconstrained attack methods, respectively.

Figures (8)

• Figure 1: The proposed Adv-CPG generates safe portraits that can deceive malicious face recognition systems. First row: original image. Second row: customized portrait based on the text prompt, without protection via Adv-CPG. Third row: customized portrait based on the scene text prompt, with protection via Adv-CPG. Fourth row: comparison with existing methods. The yellow number over each image: confidence score returned by Face++ when using the adversarial example for identity matching with the target (higher is better).
• Figure 2: The overall framework of the proposed Adv-CPG. The framework comprises three key modules: [A] a multi-modal image customizer (MMIC), [B] an ID encryptor (En1), and [C] an encryption enhancer (En2). En1 and En2 enable incremental facial privacy protection, and MMIC achieves fine-grained and personalized portrait generation.
• Figure 3: Visual quality comparison of adversarial examples generated by 4 types of adversarial methods on the CelebA-HQ dataset. The images generated by Adv-CPG implement both portrait customization and effective facial privacy protection. Red/blue (1/2) numbers below each image: confidence scores returned by Face++ and Aliyun (higher is better). Reference column: makeup reference for DiffAM.
• Figure 4: The confidence scores ($\uparrow$) returned from commercial APIs, Face++ and Aliyun. Adv-CPG has higher and more stable confidence scores than state-of-the-art noise-based and makeup-based facial privacy protection methods.
• Figure 5: Visual comparison of adversarial examples for different target identities. First column: original face. Second column: personalized portrait without protection. Third/Fourth/Fifth/Sixth column: personalized portrait with protection to target-1/2/3/4.