Safeguarding Facial Identity against Diffusion-based Face Swapping via Cascading Pathway Disruption

TL;DR

Diffusion-based face swapping poses serious privacy concerns as realistic identity manipulation becomes easier. The authors introduce VoidFace, a systemic defense that treats face swapping as a coupled identity pathway and implements cascading disruption by targeting localization, identity embedding, and generative conditioning, all optimized in a latent diffusion space with a perceptual adaptive strategy. The defense comprises four losses across physical, semantic, and generative domains and a total objective that reinforces downstream disruption, achieving superior protection while maintaining image quality and transferability to GAN-based models. Extensive experiments on CelebA-HQ and VGGFace2-HQ demonstrate robust performance under common distortions, and user studies confirm perceptual naturalness of protected faces, highlighting practical privacy-preserving potential. The approach provides a generalizable framework for safeguarding facial identity against evolving diffusion-based manipulation.

Abstract

The rapid evolution of diffusion models has democratized face swapping but also raises concerns about privacy and identity security. Existing proactive defenses, often adapted from image editing attacks, prove ineffective in this context. We attribute this failure to an oversight of the structural resilience and the unique static conditional guidance mechanism inherent in face swapping systems. To address this, we propose VoidFace, a systemic defense method that views face swapping as a coupled identity pathway. By injecting perturbations at critical bottlenecks, VoidFace induces cascading disruption throughout the pipeline. Specifically, we first introduce localization disruption and identity erasure to degrade physical regression and semantic embeddings, thereby impairing the accurate modeling of the source face. We then intervene in the generative domain by decoupling attention mechanisms to sever identity injection, and corrupting intermediate diffusion features to prevent the reconstruction of source identity. To ensure visual imperceptibility, we perform adversarial search in the latent manifold, guided by a perceptual adaptive strategy to balance attack potency with image quality. Extensive experiments show that VoidFace outperforms existing defenses across various diffusion-based swapping models, while producing adversarial faces with superior visual quality.

Figures (8)

• Figure 1: (a) Diffusion-based face swapping operates as a coupled identity pathway across multiple stages. (b) Our proposed VoidFace prevents unauthorized face swapping, safeguarding facial identity.
• Figure 2: Overview of the proposed VoidFace. We formulate the adversarial defense as a cascading disruption of the identity pathway. We systematically target critical bottlenecks across the physical, semantic, and generative domains. The perceptual adaptive latent optimization strategy is integrated to adaptively balance attack potency and visual quality of adversarial faces.
• Figure 3: Comparison of visual quality and defense effectiveness against DiffFace. The top and middle rows display protected images and details. The baselines often introduce noticeable noise, while our method generates natural faces. The bottom row shows the face swapping outputs using the protected images as source face. We effectively disrupt face swapping, leading to severe artifacts and mismatched identity.
• Figure 4: Qualitative evaluation of our protection efficacy. The evaluation covers a wide range of architectures, including four diffusion-based and two GAN-based models. As observed, our method effectively safeguards the source images, where the protected outputs suffer from severe visual degradation or complete identity mismatch, verifying both the effectiveness and transferability of our approach.
• Figure 5: User preference study. Blue bars indicate the percentage of votes favoring our method.