FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image Diffusion Models

TL;DR

FT-Shield provides copyright protection for fine-tuning text-to-image diffusion models by embedding watermarks that are quickly learned during early fine-tuning and detectable across diverse fine-tuning methods. It combines a bi-level watermark generation objective with a Mixture of Experts detector to achieve cross-method robustness, formalized by $\min_{\theta_1}\min_{\{\delta_i\}} \sum_i L_{LDM}(\cdot)$ and $D_w(x)=\sum_i \text{softmax}_i(G(x))\cdot E_i(x)$. Empirical results on art-style and object-transfer tasks show high detection rates (TPR near 100%, FPR near 0–5%) with minimal impact on image quality (FID competitive with baselines) and strong robustness under limited fine-tuning steps and corruptions when detectors are augmented. The approach outperforms prior watermarking methods like Gen-Watermark and DIAGNOSIS, offering scalable, cross-method IP protection for diffusion-based personalization without overly compromising release quality.

Abstract

Text-to-image generative models, especially those based on latent diffusion models (LDMs), have demonstrated outstanding ability in generating high-quality and high-resolution images from textual prompts. With this advancement, various fine-tuning methods have been developed to personalize text-to-image models for specific applications such as artistic style adaptation and human face transfer. However, such advancements have raised copyright concerns, especially when the data are used for personalization without authorization. For example, a malicious user can employ fine-tuning techniques to replicate the style of an artist without consent. In light of this concern, we propose FT-Shield, a watermarking solution tailored for the fine-tuning of text-to-image diffusion models. FT-Shield addresses copyright protection challenges by designing new watermark generation and detection strategies. In particular, it introduces an innovative algorithm for watermark generation. It ensures the seamless transfer of watermarks from training images to generated outputs, facilitating the identification of copyrighted material use. To tackle the variability in fine-tuning methods and their impact on watermark detection, FT-Shield integrates a Mixture of Experts (MoE) approach for watermark detection. Comprehensive experiments validate the effectiveness of our proposed FT-Shield.

Figures (7)

• Figure 1: An illustration of generated images from fine-tuned text-to-image models w.r.t. fine-tuning steps. While previous work ma2023generative requires extensive fine-tuning to ensure that the watermark is learned, our method enables the watermark to be learned in the early stages of fine-tuning. Prompt used for the generation: Cherry blossoms in full bloom. Targeted style: the style of artist Beihong Xu.
• Figure 2: An overview of the two-stage watermarking protection process
• Figure 3: An illustration of mixture of watermark detectors.
• Figure 4: Examples of watermarked images (first line) and generated images (other lines) in the style of artist Beihong Xu. The prompt of generation: A frog on a lotus Leaf.
• Figure 5: Examples of watermarked images (first line) and images generated through domain adaptation for Pokemon imagery (other lines). The prompt of generation: A robotic cat with wings.