Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning

Jonathan Lebensold, Maziar Sanjabi, Pietro Astolfi, Adriana Romero-Soriano, Kamalika Chaudhuri, Mike Rabbat, Chuan Guo

TL;DR

DP-RDM addresses the privacy risks of retrieval-augmented diffusion by introducing a differentially private retrieval mechanism that augments prompts with privatized samples without requiring fine-tuning. The approach combines private $k$-NN retrieval with noisy aggregation and query interpolation to balance privacy and image quality, backed by a Rényi-DP analysis and practical DP guarantees for multiple queries. Empirically, DP-RDM achieves competitive quality under DP on CIFAR-10, MS-COCO, and Shutterstock, with notable gains on MS-COCO (e.g., $\epsilon=10$ and FID improvements from $14.4$ to $10.9$) as the private retrieval dataset scales. The work demonstrates that large-scale private retrieval can enable domain adaptation for diffusion models while maintaining rigorous privacy, potentially broadening privacy-preserving deployment of generative systems in sensitive domains.

Abstract

Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $ε=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries.

DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning

TL;DR

DP-RDM addresses the privacy risks of retrieval-augmented diffusion by introducing a differentially private retrieval mechanism that augments prompts with privatized samples without requiring fine-tuning. The approach combines private -NN retrieval with noisy aggregation and query interpolation to balance privacy and image quality, backed by a Rényi-DP analysis and practical DP guarantees for multiple queries. Empirically, DP-RDM achieves competitive quality under DP on CIFAR-10, MS-COCO, and Shutterstock, with notable gains on MS-COCO (e.g., and FID improvements from to ) as the private retrieval dataset scales. The work demonstrates that large-scale private retrieval can enable domain adaptation for diffusion models while maintaining rigorous privacy, potentially broadening privacy-preserving deployment of generative systems in sensitive domains.

Abstract

Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of , while providing a point improvement in FID compared to public-only retrieval for up to queries.
Paper Structure (45 sections, 2 theorems, 8 equations, 24 figures, 1 algorithm)

This paper contains 45 sections, 2 theorems, 8 equations, 24 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background
  4. Diffusion Models and Retrieval Augmented Generation
  5. Differential Privacy
  6. DP mechanisms.
  7. DP training/fine-tuning.
  8. Differentially Private RDM
  9. Motivation: Privacy Risks in Retrieval Augmented Generation
  10. DP-RDM Architecture
  11. Retrieval dataset construction.
  12. k-NN retrieval with random subset selection.
  13. Noisy aggregation.
  14. Query interpolation.
  15. Model Training
  16. ...and 30 more sections

Key Result

Theorem 2.6

Let $\text{SG}_{q, \sigma}(D)$ be defined as def:sgm for function $f$. Then $\text{SG}_{q, \sigma}(D)$ satisfies $(\alpha, \epsilon)$-RDP with if $\| f(D) - f(D') \|_2 \leq 1$ for adjacent $D, D' \in \cal{S}$.

Figures (24)

  • Figure 1: Samples generated by our differentially private retrieval-augmented diffusion model (DP-RDM), which was trained on face-blurred ImageNet, using different private retrieval datasets at inference time: Shutterstock, MS-COCO with face-blurring (FB), ImageNet FB, and CIFAR-10. We calibrated the noise added in each row for a privacy budget of $\epsilon = 10$ after answering $1,000$ queries. Each query uses $k=18$ neighbors for retrieval augmentation and a 0.1%-0.3% random subset of the retrieval dataset. The differences between the generated images show how swapping the private retrieval dataset changes the distribution of the generated images to adapt to a given data domain, e.g., the differences in laptop and smart watch, or missing concepts such as goose, llama, smart watch and laptop in CIFAR-10.
  • Figure 2: (a) RDM architecture from blattmann2022retrieval. (b) Samples generated with a non-private RDM. The retrieval dataset consists of blank images and one illustration of the Eiffel Tower with a Shutterstock watermark. Each row shows samples for a different number of retrieved neighbors $k$. The watermark is clearly visible even though it came only from conditioning on the retrieval dataset.
  • Figure 3: Text-to-image generation with DP-RDM using a private retrieval dataset. Yellow blocks refer to models trained on public datasets. The private $k$-NN block, denoted in pink, illustrates the privacy boundary between public and private data.
  • Figure 4: Pseudo-code description of the private image generation procedure.
  • Figure 5: Privacy loss $\epsilon$ for generating $1,000$ images of certain concepts with concept density $r$ in log-log scale. For concepts with high concept density, it is possible to generate a large number of high-quality images under low privacy cost.
  • ...and 19 more figures

Theorems & Definitions (8)

  • Definition 2.1: DP; dwork2006calibrating
  • Definition 2.2: Rényi divergence; renyi1961measures
  • Definition 2.3: Rényi DP; mironov2017renyi
  • Definition 2.4: Gaussian Mechanism
  • Definition 2.5: SGM; mironov2019r
  • Theorem 2.6: SGM satisfies RDP; mironov2019r
  • Theorem 3.1
  • proof