Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting

Bao Q. Tran, Viet Nguyen, Anh Tran, Toan Tran

TL;DR

Experiments reveal that DistillMD not only defends against MIAs but also mitigates model memorization, indicating that both vulnerabilities stem from overfitting and can be addressed simultaneously with the unified approach.

Abstract

Diffusion models have demonstrated remarkable capabilities in image synthesis, but their recently proven vulnerability to Membership Inference Attacks (MIAs) poses a critical privacy concern. This paper introduces two novel and efficient approaches (DualMD and DistillMD) to protect diffusion models against MIAs while maintaining high utility. Both methods are based on training two separate diffusion models on disjoint subsets of the original dataset. DualMD then employs a private inference pipeline that utilizes both models. This strategy significantly reduces the risk of black-box MIAs by limiting the information any single model contains about individual training samples. The dual models can also generate "soft targets" to train a private student model in DistillMD, enhancing privacy guarantees against all types of MIAs. Extensive evaluations of DualMD and DistillMD against state-of-the-art MIAs across various datasets in white-box and black-box settings demonstrate their effectiveness in substantially reducing MIA success rates while preserving competitive image generation performance. Notably, our experiments reveal that DistillMD not only defends against MIAs but also mitigates model memorization, indicating that both vulnerabilities stem from overfitting and can be addressed simultaneously with our unified approach.

Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting

TL;DR

Experiments reveal that DistillMD not only defends against MIAs but also mitigates model memorization, indicating that both vulnerabilities stem from overfitting and can be addressed simultaneously with the unified approach.

Abstract

Diffusion models have demonstrated remarkable capabilities in image synthesis, but their recently proven vulnerability to Membership Inference Attacks (MIAs) poses a critical privacy concern. This paper introduces two novel and efficient approaches (DualMD and DistillMD) to protect diffusion models against MIAs while maintaining high utility. Both methods are based on training two separate diffusion models on disjoint subsets of the original dataset. DualMD then employs a private inference pipeline that utilizes both models. This strategy significantly reduces the risk of black-box MIAs by limiting the information any single model contains about individual training samples. The dual models can also generate "soft targets" to train a private student model in DistillMD, enhancing privacy guarantees against all types of MIAs. Extensive evaluations of DualMD and DistillMD against state-of-the-art MIAs across various datasets in white-box and black-box settings demonstrate their effectiveness in substantially reducing MIA success rates while preserving competitive image generation performance. Notably, our experiments reveal that DistillMD not only defends against MIAs but also mitigates model memorization, indicating that both vulnerabilities stem from overfitting and can be addressed simultaneously with our unified approach.

Paper Structure

This paper contains 30 sections, 14 equations, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Diffusion Models
  4. Membership Inference Attacks
  5. Membership Inference Defenses
  6. Diffusion Memorization and Mitigation
  7. Methodology
  8. Membership Inference Attacks (MIAs) and Defenses
  9. Disjoint Training
  10. Alternating Distillation (DistillMD)
  11. Choosing teacher models
  12. Distillation loss
  13. Self-correcting Inference Pipeline (DualMD)
  14. Motivation
  15. Enhancing Privacy for Conditional Diffusion Models
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Our proposed defense method DistillMD with model distillation. Two teacher models are trained on two disjoint subsets with standard diffusion models training (Left). A student model is then distilled from the two teachers using the distillation loss. Only one teacher model is used to produce the target noise given one data sample, and the teacher is chosen so that the input noisy image did not appear at its training subset in the previous phase (Right).
  • Figure 2: The efficient defense method DualMD with modified inference pipeline. The two models, which are trained on disjointed subsets, are used to denoise images alternately.
  • Figure 3: Distribution of the values of the loss in Eq. \ref{['eq:mem_loss']} between 500 memorized prompts and 500 non-memorized prompts on Stable Diffusion v1.5 ldm.