DiffMark: Diffusion-based Robust Watermark Against Deepfakes

TL;DR

DiffMark tackles the vulnerability of facial image watermarks to Deepfake manipulation by introducing a diffusion-based proactive watermarking framework. It conditions the diffusion process on both the facial image and a binary watermark, with a scaled facial cue $x_c = \sqrt{1-\overline{\alpha}_t} x_0$ to preserve guidance early in sampling and gradually attenuate it. Key contributions include a cross information fusion module with positional-bit embedding indices, a training-time frozen autoencoder to simulate Deepfakes, and a Deepfake-resistant guidance mechanism during DDIM sampling to improve robustness. Experiments on CelebA-HQ and LFW show DiffMark achieves strong invisibility (high PSNR/SSIM, low LPIPS) and robust watermark recovery (low BER) under diverse benign distortions and Deepfake attacks, supporting practical use for authenticity verification and source tracing.

Abstract

Deepfakes pose significant security and privacy threats through malicious facial manipulations. While robust watermarking can aid in authenticity verification and source tracking, existing methods often lack the sufficient robustness against Deepfake manipulations. Diffusion models have demonstrated remarkable performance in image generation, enabling the seamless fusion of watermark with image during generation. In this study, we propose a novel robust watermarking framework based on diffusion model, called DiffMark. By modifying the training and sampling scheme, we take the facial image and watermark as conditions to guide the diffusion model to progressively denoise and generate corresponding watermarked image. In the construction of facial condition, we weight the facial image by a timestep-dependent factor that gradually reduces the guidance intensity with the decrease of noise, thus better adapting to the sampling process of diffusion model. To achieve the fusion of watermark condition, we introduce a cross information fusion (CIF) module that leverages a learnable embedding table to adaptively extract watermark features and integrates them with image features via cross-attention. To enhance the robustness of the watermark against Deepfake manipulations, we integrate a frozen autoencoder during training phase to simulate Deepfake manipulations. Additionally, we introduce Deepfake-resistant guidance that employs specific Deepfake model to adversarially guide the diffusion sampling process to generate more robust watermarked images. Experimental results demonstrate the effectiveness of the proposed DiffMark on typical Deepfakes. Our code will be available at https://github.com/vpsg-research/DiffMark.

Figures (4)

• Figure 1: The difference between our method and the existing methods: (a) Traditional pixel-space methods directly embed the watermark in the pixel space of the image; (b) Latent-space methods that transform image into latent representation for watermark embedding; (c) Our method initiates from standard Gaussian distribution, using facial image and watermark as conditions to guide the diffusion model denoising for watermarked image generation.
• Figure 2: Illustration of the proposed DiffMark. (a) Training Phase: The t-step noised image $x_t$, dynamically scaled facial image $\sqrt{1 - \overline{\alpha}_t} x_0$, watermark $w$ and timestep $t$ are fed into the diffusion encoder to predict the watermarked image $\hat{x}_0$, which is then reconstructed by a frozen autoencoder to produce $\widetilde{x}_0$. The watermark decoder extracts the watermark from $\widetilde{x}_0$. (b) Inference Phase: Initialized with standard Gaussian distribution $x_T$, the Deepfake-resistant guided DDIM sampling process take the scaled facial image $x_c$ and watermark $w$ as conditions to gradually denoise and generate the watermarked image. The watermark can be extracted by the watermark decoder from the distorted image for source tracing.
• Figure 3: Cross information fusion module. This module combines a learnable embedding table with a cross-attention mechanism.
• Figure 4: The visual quality of facial images under various typical distortions. The rows from top to bottom display: (a) the original cover image $x_\text{co}$, (b) the watermarked image $x_\text{wm}$, (c) the distorted image $x_\text{dt}$, (d) the normalized residual signal between $x_\text{wm}$ and $x_\text{co}$, and (e) the normalized residual signal between $x_\text{dt}$ and $x_\text{wm}$. Each column represents a distinct distortion type. All images have a size of $256\times256$ pixels.