Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy

Shengfang Zhai, Huanran Chen, Yinpeng Dong, Jiajun Li, Qingni Shen, Yansong Gao, Hang Su, Yang Liu

TL;DR

This paper identifies a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the corresponding text rather than the marginal distribution of images only, and derives an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference, which reduces the stochasticity in estimating memorization of individual samples.

Abstract

Text-to-image diffusion models have achieved tremendous success in the field of controllable image generation, while also coming along with issues of privacy leakage and data copyrights. Membership inference arises in these contexts as a potential auditing method for detecting unauthorized data usage. While some efforts have been made on diffusion models, they are not applicable to text-to-image diffusion models due to the high computation overhead and enhanced generalization capabilities. In this paper, we first identify a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the corresponding text rather than the marginal distribution of images only. Based on this observation, we derive an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference, which reduces the stochasticity in estimating memorization of individual samples. Experimental results demonstrate that our method significantly outperforms previous methods across various data distributions and dataset scales. Additionally, our method shows superior resistance to overfitting mitigation strategies, such as early stopping and data augmentation.

Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy

TL;DR

This paper identifies a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the corresponding text rather than the marginal distribution of images only, and derives an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference, which reduces the stochasticity in estimating memorization of individual samples.

Abstract

Text-to-image diffusion models have achieved tremendous success in the field of controllable image generation, while also coming along with issues of privacy leakage and data copyrights. Membership inference arises in these contexts as a potential auditing method for detecting unauthorized data usage. While some efforts have been made on diffusion models, they are not applicable to text-to-image diffusion models due to the high computation overhead and enhanced generalization capabilities. In this paper, we first identify a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the corresponding text rather than the marginal distribution of images only. Based on this observation, we derive an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference, which reduces the stochasticity in estimating memorization of individual samples. Experimental results demonstrate that our method significantly outperforms previous methods across various data distributions and dataset scales. Additionally, our method shows superior resistance to overfitting mitigation strategies, such as early stopping and data augmentation.
Paper Structure (25 sections, 1 theorem, 27 equations, 6 figures, 6 tables)

This paper contains 25 sections, 1 theorem, 27 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Diffusion Model Preliminaries
  3. Methodology
  4. Threat Model
  5. Conditional Overfitting Phenomenon
  6. Condition Likelihood Discrepancy
  7. Implementation of CLiD-MI
  8. Practical Considerations
  9. Experiments
  10. Setups
  11. Main Results
  12. Performance on Various Training Steps
  13. Ablation Study
  14. Resistance to Defense
  15. Weaker Assumption
  16. ...and 10 more sections

Key Result

Theorem 3.2

(Proof in Appendix appd: proof) When using $D=D_{KL}$ as distance metric, Assumption theorem:assumption is equivalent to: where

Figures (6)

  • Figure 1: FID values and the FID differences of synthetic images ($2500/2500$ samples for member/hold-out set) under different conditions of member set and hold-out set.
  • Figure 2: The performance of membership inference methods on Stable Diffusion v1-5 Stable-Diffusion-v1-5 in pretraining setting. We utilize the processed LAION dataset to ensure the distribution consistency between member / hold-out sets dubinski2024towardsdas2024blind. The best results are highlighted in bold.
  • Figure 3: Effectiveness trajectory on various training steps.
  • Figure 4: Performance of $\text{CLiD}_{th}$ and SecMI under various Monte Carlo sampling numbers (i.e., query count). The legend labels are sorted in ascending order by AUC values.
  • Figure A.1: Metric values and the metric differences of synthetic images, with the same setting as Sec. \ref{['sec:key_i']}.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 3.2
  • proof