Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping

Hon Ming Yam, Zhongliang Guo, Chun Pong Lau

TL;DR

This work addresses the privacy risks posed by diffusion-based face swapping by proposing a proactive adversarial defense. It introduces a latent-space perturbation framework that combines a face identity loss with an inference-step averaging technique to produce robust, memory-efficient protections against diffusion models. The method demonstrates robustness across diverse diffusion-based face swapping pipelines and against purification defenses, while preserving image quality. Experimental results on CelebA-HQ with StableDiffusion and ArcFace indicate substantial disruption of identity transfer and transferability to unseen models like REFace, underscoring practical benefits for facial privacy protection.

Abstract

The proliferation of diffusion-based deepfake technologies poses significant risks for unauthorized and unethical facial image manipulation. While traditional countermeasures have primarily focused on passive detection methods, this paper introduces a novel proactive defense strategy through adversarial attacks that preemptively protect facial images from being exploited by diffusion-based deepfake systems. Existing adversarial protection methods predominantly target conventional generative architectures (GANs, AEs, VAEs) and fail to address the unique challenges presented by diffusion models, which have become the predominant framework for high-quality facial deepfakes. Current diffusion-specific adversarial approaches are limited by their reliance on specific model architectures and weights, rendering them ineffective against the diverse landscape of diffusion-based deepfake implementations. Additionally, they typically employ global perturbation strategies that inadequately address the region-specific nature of facial manipulation in deepfakes.

My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping

TL;DR

This work addresses the privacy risks posed by diffusion-based face swapping by proposing a proactive adversarial defense. It introduces a latent-space perturbation framework that combines a face identity loss with an inference-step averaging technique to produce robust, memory-efficient protections against diffusion models. The method demonstrates robustness across diverse diffusion-based face swapping pipelines and against purification defenses, while preserving image quality. Experimental results on CelebA-HQ with StableDiffusion and ArcFace indicate substantial disruption of identity transfer and transferability to unseen models like REFace, underscoring practical benefits for facial privacy protection.

Abstract

The proliferation of diffusion-based deepfake technologies poses significant risks for unauthorized and unethical facial image manipulation. While traditional countermeasures have primarily focused on passive detection methods, this paper introduces a novel proactive defense strategy through adversarial attacks that preemptively protect facial images from being exploited by diffusion-based deepfake systems. Existing adversarial protection methods predominantly target conventional generative architectures (GANs, AEs, VAEs) and fail to address the unique challenges presented by diffusion models, which have become the predominant framework for high-quality facial deepfakes. Current diffusion-specific adversarial approaches are limited by their reliance on specific model architectures and weights, rendering them ineffective against the diverse landscape of diffusion-based deepfake implementations. Additionally, they typically employ global perturbation strategies that inadequately address the region-specific nature of facial manipulation in deepfakes.

Paper Structure

This paper contains 29 sections, 14 equations, 10 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attack
  4. Diffusion Model
  5. Face Swapping Methods Based on Deep Learning
  6. Adversarial Attack on Face Swapping Models
  7. Adversarial Attack on Diffusion Model
  8. Preliminary
  9. Diffusion model
  10. Conditional Diffusion for Face Swapping
  11. Classifier-Free Guidance
  12. SDEdit
  13. Method
  14. Latent Space Perturbation
  15. Identity Loss (L_ID)
  16. ...and 14 more sections

Figures (10)

  • Figure 1: The face swapping result using diffusion-based model pre- and post-protection by our method and other existing methods.
  • Figure 2: The pipeline of our method comparing to conventional gradient-based method
  • Figure 3: qualitative demonstration of our method comparing with SDS, AdvDM, SDST, DiffusionGuard, PhotoGuard across different diffusion-based face swapping techniques. The top row is face swapping result conducted on Face Adapter, whereas the bottom row is face swapping result conducted on REFace.
  • Figure 4: A qualitative visualization of our method comparing with existing methods on Face-Adapter. It is worth noting that, DG refers to DiffusionGuard, while PG refers to PhotoGuard.
  • Figure 5: A qualitative visualization of our method comparing with existing methods on REFace. It is worth noting that, DG refers to DiffusionGuard, while PG refers to PhotoGuard.
  • ...and 5 more figures