Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forging and Removing Latent-Noise Diffusion Watermarks Using a Single Image

Anubhav Jain, Yuya Kobayashi, Naoki Murata, Yuhta Takida, Takashi Shibuya, Yuki Mitsufuji, Niv Cohen, Nasir Memon, Julian Togelius

TL;DR

This work reveals a latent-region vulnerability in diffusion-model watermarks by showing that a region in the clean latent space $Z_0^{(w)}(\mathcal{W},k)$ consistently maps to a watermark key via DDIM inversion. The authors propose a black-box attack that requires only a single watermarked image and a proxy VAE to forge or remove watermarks by perturbing an image so that its latent representation aligns with or diverges from the watermark region. They formalize both forgery and removal objectives with explicit loss formulations and validate the approach across four latent-noise watermark schemes (Tree-Ring, RingID, WIND, Gaussian Shading) on SDv1.4 and SDv2.0, achieving high attack success rates while controlling perceptual distortion. The findings highlight practical vulnerabilities in current watermarking methods and motivate the development of more robust watermarking strategies that do not rely solely on the initial-noise embedding. Overall, the work emphasizes the need for stronger protections against adversarial manipulation in diffusion-model provenance techniques.

Abstract

Watermarking techniques are vital for protecting intellectual property and preventing fraudulent use of media. Most previous watermarking schemes designed for diffusion models embed a secret key in the initial noise. The resulting pattern is often considered hard to remove and forge into unrelated images. In this paper, we propose a black-box adversarial attack without presuming access to the diffusion model weights. Our attack uses only a single watermarked example and is based on a simple observation: there is a many-to-one mapping between images and initial noises. There are regions in the clean image latent space pertaining to each watermark that get mapped to the same initial noise when inverted. Based on this intuition, we propose an adversarial attack to forge the watermark by introducing perturbations to the images such that we can enter the region of watermarked images. We show that we can also apply a similar approach for watermark removal by learning perturbations to exit this region. We report results on multiple watermarking schemes (Tree-Ring, RingID, WIND, and Gaussian Shading) across two diffusion models (SDv1.4 and SDv2.0). Our results demonstrate the effectiveness of the attack and expose vulnerabilities in the watermarking methods, motivating future research on improving them.

Forging and Removing Latent-Noise Diffusion Watermarks Using a Single Image

TL;DR

This work reveals a latent-region vulnerability in diffusion-model watermarks by showing that a region in the clean latent space consistently maps to a watermark key via DDIM inversion. The authors propose a black-box attack that requires only a single watermarked image and a proxy VAE to forge or remove watermarks by perturbing an image so that its latent representation aligns with or diverges from the watermark region. They formalize both forgery and removal objectives with explicit loss formulations and validate the approach across four latent-noise watermark schemes (Tree-Ring, RingID, WIND, Gaussian Shading) on SDv1.4 and SDv2.0, achieving high attack success rates while controlling perceptual distortion. The findings highlight practical vulnerabilities in current watermarking methods and motivate the development of more robust watermarking strategies that do not rely solely on the initial-noise embedding. Overall, the work emphasizes the need for stronger protections against adversarial manipulation in diffusion-model provenance techniques.

Abstract

Watermarking techniques are vital for protecting intellectual property and preventing fraudulent use of media. Most previous watermarking schemes designed for diffusion models embed a secret key in the initial noise. The resulting pattern is often considered hard to remove and forge into unrelated images. In this paper, we propose a black-box adversarial attack without presuming access to the diffusion model weights. Our attack uses only a single watermarked example and is based on a simple observation: there is a many-to-one mapping between images and initial noises. There are regions in the clean image latent space pertaining to each watermark that get mapped to the same initial noise when inverted. Based on this intuition, we propose an adversarial attack to forge the watermark by introducing perturbations to the images such that we can enter the region of watermarked images. We show that we can also apply a similar approach for watermark removal by learning perturbations to exit this region. We report results on multiple watermarking schemes (Tree-Ring, RingID, WIND, and Gaussian Shading) across two diffusion models (SDv1.4 and SDv2.0). Our results demonstrate the effectiveness of the attack and expose vulnerabilities in the watermarking methods, motivating future research on improving them.
Paper Structure (32 sections, 8 equations, 18 figures, 6 tables)

This paper contains 32 sections, 8 equations, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Watermarking Schemes for Diffusion Models.
  4. Forgery and Removal Attacks against Watermarks.
  5. Preliminaries
  6. Watermarking Scheme in Diffusion Models
  7. Watermarking Attack Techniques
  8. Threat Model
  9. Motivation
  10. Imperceptible Attack Against Watermarks
  11. Forgery Attack.
  12. Removal Attack.
  13. Experimental Results
  14. Experimental Setup.
  15. Evaluation Metrics.
  16. ...and 17 more sections

Figures (18)

  • Figure 1: Intuition behind the attack: A whole region exists in the clean latent space, all of which approximately maps to a single key-embedded initial latent noise vector. A forgerer only needs to ensure their sample embeds within this region to be falsely classified as watermarked.
  • Figure 2: Our forgery attack works by finding an adversarial perturbation $\bm{\delta}$ such that the latent representation of the non-watermarked image $\mathcal{E}_{\phi}(\mathbf{x}^{(c)} + \bm{\delta})$ is close to the one corresponding to that of a watermarked image $\mathcal{E}_{\phi}(\mathbf{x}^{(w)})$. We do so while ensuring that we only introduce imperceptible changes to the clean image.
  • Figure 3: Motivation for the attack. There exists a latent direction/region pertaining to watermarked latents from a specific secret key in the clean image latent space. The further we traverse in the relevant direction, the stronger the attack becomes. Our method exploits this vulnerability while better preserving the image content. See Appendix Figure \ref{['fig:directions_appendix']} for more examples.
  • Figure 4: Two-dimensional visualization of the latent space showing the linear separability of watermarked and non-watermarked images. The horizontal axis is obtained by linear discriminant analysis (LDA), while the vertical axis is a random projection.
  • Figure 5: Examples showing successful watermark forgery attacks on the Tree-Ring watermarking method with different hyperparameter $\lambda$ values. The hyperparameter $\lambda$ controls the trade-off between ASR and the amount of perturbations we introduce.
  • ...and 13 more figures

Theorems & Definitions (1)

  • definition 1: Watermark Region