Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage

TL;DR

This paper proposes RATTAN, a text-to-image diffusion model, that leverages the diffusion process to conduct controlled image generation on the protected input, preserving the high-level features of the input while ignoring the low-level details utilized by watermarks.

Abstract

Text-to-image diffusion models, such as Stable Diffusion, have shown exceptional potential in generating high-quality images. However, recent studies highlight concerns over the use of unauthorized data in training these models, which may lead to intellectual property infringement or privacy violations. A promising approach to mitigate these issues is to apply a watermark to images and subsequently check if generative models reproduce similar watermark features. In this paper, we examine the robustness of various watermark-based protection methods applied to text-to-image models. We observe that common image transformations are ineffective at removing the watermark effect. Therefore, we propose RATTAN, that leverages the diffusion process to conduct controlled image generation on the protected input, preserving the high-level features of the input while ignoring the low-level details utilized by watermarks. A small number of generated images are then used to fine-tune protected models. Our experiments on three datasets and 140 text-to-image diffusion models reveal that existing state-of-the-art protections are not robust against RATTAN.

Figures (7)

• Figure 1: The top part represents the existing watermarking procedure for text-to-image diffusion models. The bottom part illustrates our method, Rattan, for bypassing watermark-based protections. The text-to-image diffusion models shown in black, blue, and green, denote the off-the-shelf base, watermarked, and watermark-free versions, respectively.
• Figure 2: Comparison of (a) the original image, (b) DIAGNOSIS watermarked image, and the images after applying (c) Color Jittering, (d) Gaussian Blur, (e) JPEG Compression. The bottom row provides a zoomed-in view. The curly-line characteristic of the watermark is still visible in each transformed image. (f) presents the result after Rattan's controlled image generation on the watermarked input. The lines appear smoother compared to the original image.
• Figure 3: Comparison of images generated by different versions of Stable Diffusion.
• Figure 4: Intermediate images during controlled image generation of Rattan with $\gamma=0.6$.
• Figure 5: Intermediate images during controlled image generation of Rattan with $\gamma=1.0$.