Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis

Songrui Wang, Yubo Zhu, Wei Tong, Sheng Zhong

TL;DR

This paper presents a dataset watermarking framework designed to detect unauthorized usage and trace data leaks, and demonstrates the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.

Abstract

Text-to-image synthesis has become highly popular for generating realistic and stylized images, often requiring fine-tuning generative models with domain-specific datasets for specialized tasks. However, these valuable datasets face risks of unauthorized usage and unapproved sharing, compromising the rights of the owners. In this paper, we address the issue of dataset abuse during the fine-tuning of Stable Diffusion models for text-to-image synthesis. We present a dataset watermarking framework designed to detect unauthorized usage and trace data leaks. The framework employs two key strategies across multiple watermarking schemes and is effective for large-scale dataset authorization. Extensive experiments demonstrate the framework's effectiveness, minimal impact on the dataset (only 2% of the data required to be modified for high detection accuracy), and ability to trace data leaks. Our results also highlight the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.

Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis

TL;DR

This paper presents a dataset watermarking framework designed to detect unauthorized usage and trace data leaks, and demonstrates the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.

Abstract

Text-to-image synthesis has become highly popular for generating realistic and stylized images, often requiring fine-tuning generative models with domain-specific datasets for specialized tasks. However, these valuable datasets face risks of unauthorized usage and unapproved sharing, compromising the rights of the owners. In this paper, we address the issue of dataset abuse during the fine-tuning of Stable Diffusion models for text-to-image synthesis. We present a dataset watermarking framework designed to detect unauthorized usage and trace data leaks. The framework employs two key strategies across multiple watermarking schemes and is effective for large-scale dataset authorization. Extensive experiments demonstrate the framework's effectiveness, minimal impact on the dataset (only 2% of the data required to be modified for high detection accuracy), and ability to trace data leaks. Our results also highlight the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.
Paper Structure (74 sections, 4 equations, 26 figures, 16 tables, 4 algorithms)

This paper contains 74 sections, 4 equations, 26 figures, 16 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Text-to-Image Synthesis
  4. Generative Model Fine-Tuning
  5. Problem Statement
  6. Threat Model
  7. Design Goals
  8. Straightforward Approaches: Infeasibility
  9. Framework
  10. Overview
  11. Activation Token Selection and Watermark Injection
  12. Watermark Detection
  13. Design Details
  14. Image Watermarking Schemes
  15. Activation Token Selection
  16. ...and 59 more sections

Figures (26)

  • Figure 1: Dataset authorization and potential abuse in fine-tuning text-to-image models. Alice, the data user, requests access to the dataset from the owner. The owner embeds the ownership information and then authorizes Alice to use it. A malicious user may get the dataset without authorization from the owner. The owner should be able to detect any unauthorized use of the data and identify the source of the leak.
  • Figure 2: The proposed dataset watermarking framework.
  • Figure 3: Examples from the WikiArt and COCO datasets before and after watermarking.
  • Figure 4: TWA: detection accuracy w/ various injection ratios
  • Figure 5: WAA: detection accuracy w/ various training epochs
  • ...and 21 more figures