Beauty and the Beast: Imperceptible Perturbations Against Diffusion-Based Face Swapping via Directional Attribute Editing

TL;DR

FaceDefense tackles the risk of diffusion-based face swapping by generating imperceptible, latent-space adversarial perturbations that degrade swapped outputs while preserving image quality. It combines a diffusion-aware adversarial loss with directional multi-attribute editing to restore perturbation-induced distortions, implemented through a two-phase alternating optimization between editing and perturbation generation. The approach outperforms prior latent- and pixel-space defenses on multiple datasets, maintaining robustness against common image processing and exhibiting transferability to several diffusion-based swapping models. This provides a practical, user-friendly defense that preserves privacy without sacrificing visual fidelity. The work highlights the potential and limitations of proactive defenses in real-world social-media contexts and points to future improvements in speed and editing-space integration.

Abstract

Diffusion-based face swapping achieves state-of-the-art performance, yet it also exacerbates the potential harm of malicious face swapping to violate portraiture right or undermine personal reputation. This has spurred the development of proactive defense methods. However, existing approaches face a core trade-off: large perturbations distort facial structures, while small ones weaken protection effectiveness. To address these issues, we propose FaceDefense, an enhanced proactive defense framework against diffusion-based face swapping. Our method introduces a new diffusion loss to strengthen the defensive efficacy of adversarial examples, and employs a directional facial attribute editing to restore perturbation-induced distortions, thereby enhancing visual imperceptibility. A two-phase alternating optimization strategy is designed to generate final perturbed face images. Extensive experiments show that FaceDefense significantly outperforms existing methods in both imperceptibility and defense effectiveness, achieving a superior trade-off.

Figures (16)

• Figure 1: Comparison between the representative method Myfaceyam2025my and our method. “*” denotes adversarial examples, and “#” denotes face swapping results obtained from adversarial examples. Regardless of the $eps$ value, our method exhibits stronger defense effectiveness and imperceptibility.
• Figure 2: The overall flowchart of FaceDefense, where “Swap” denotes the malicious face-swapping operation, “Target Network” refers to the LDMs that requires defense, “Attribute Editing” indicates the module with directional multi-attribute editing function.
• Figure 3: Evaluation curves across metrics; each marker denotes one experiment. XM2, Vox, and Cele refer to the XM2VTS, VoxCeleb2, and CelebA-HQ datasets, respectively. All curves are interpolated and smoothed. In (a), curves closer to the top-left corner indicate stronger defense performance. For (b)-(d), metrics favoring lower values have been negated (higher is better) on the y-axes of (b) and (c), and the x-axis of (d), so curves near the top-right corner representing the best trade-off between visual quality and defense efficacy.
• Figure 4: Defense results of our adversarial examples on the DiffSwap zhaoDiffSwap2023a model.
• Figure 5: The $\mathcal{L}_{attack}$ curve for the image “bengio.jpg” under joint optimization. The number on the every image denotes the current iteration round. We extract the adversarial example at current round and convert it to RGB space for visualization.