Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CPR: Retrieval Augmented Generation for Copyright Protection

Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, Stefano Soatto

TL;DR

This work introduces Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models, and proves that CPR satisfies Near Access Freeness (NAF), which bounds the amount of information an attacker may be able to extract from the generated images.

Abstract

Retrieval Augmented Generation (RAG) is emerging as a flexible and robust technique to adapt models to private users data without training, to handle credit attribution, and to allow efficient machine unlearning at scale. However, RAG techniques for image generation may lead to parts of the retrieved samples being copied in the model's output. To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models.CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs. In particular, it does so by sampling from a mixture of public (safe) distribution and private (user) distribution by merging their diffusion scores at inference. We prove that CPR satisfies Near Access Freeness (NAF) which bounds the amount of information an attacker may be able to extract from the generated images. We provide two algorithms for copyright protection, CPR-KL and CPR-Choose. Unlike previously proposed rejection-sampling-based NAF methods, our methods enable efficient copyright-protected sampling with a single run of backward diffusion. We show that our method can be applied to any pre-trained conditional diffusion model, such as Stable Diffusion or unCLIP. In particular, we empirically show that applying CPR on top of unCLIP improves quality and text-to-image alignment of the generated results (81.4 to 83.17 on TIFA benchmark), while enabling credit attribution, copy-right protection, and deterministic, constant time, unlearning.

CPR: Retrieval Augmented Generation for Copyright Protection

TL;DR

This work introduces Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models, and proves that CPR satisfies Near Access Freeness (NAF), which bounds the amount of information an attacker may be able to extract from the generated images.

Abstract

Retrieval Augmented Generation (RAG) is emerging as a flexible and robust technique to adapt models to private users data without training, to handle credit attribution, and to allow efficient machine unlearning at scale. However, RAG techniques for image generation may lead to parts of the retrieved samples being copied in the model's output. To reduce risks of leaking private information contained in the retrieved set, we introduce Copy-Protected generation with Retrieval (CPR), a new method for RAG with strong copyright protection guarantees in a mixed-private setting for diffusion models.CPR allows to condition the output of diffusion models on a set of retrieved images, while also guaranteeing that unique identifiable information about those example is not exposed in the generated outputs. In particular, it does so by sampling from a mixture of public (safe) distribution and private (user) distribution by merging their diffusion scores at inference. We prove that CPR satisfies Near Access Freeness (NAF) which bounds the amount of information an attacker may be able to extract from the generated images. We provide two algorithms for copyright protection, CPR-KL and CPR-Choose. Unlike previously proposed rejection-sampling-based NAF methods, our methods enable efficient copyright-protected sampling with a single run of backward diffusion. We show that our method can be applied to any pre-trained conditional diffusion model, such as Stable Diffusion or unCLIP. In particular, we empirically show that applying CPR on top of unCLIP improves quality and text-to-image alignment of the generated results (81.4 to 83.17 on TIFA benchmark), while enabling credit attribution, copy-right protection, and deterministic, constant time, unlearning.
Paper Structure (36 sections, 6 theorems, 25 equations, 7 figures, 1 table, 2 algorithms)

This paper contains 36 sections, 6 theorems, 25 equations, 7 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Retrieval Augmented Generation
  4. Image Manipulation
  5. Privacy
  6. Preliminaries
  7. Mixed-Privacy RAG
  8. Retrieval
  9. Mixture-of-Distribution
  10. Mixture-of-Score
  11. Copy-Protected Generation
  12. Near-Access Freeness
  13. CPR-KL
  14. CPR-Choose
  15. Time-Discretization
  16. ...and 21 more sections

Key Result

Proposition 1

Let $p_t(x_t|c)$ be as in eq:mixture-at-t, then $\nabla_{x_t}\log p_t(x_t|c)$ is given by: where we have defined: and $p^t_D(x_t|c)$ denotes the forward flow of the distribution $p_D(x_t|c)$ at time $t$ (and similarly for $p^t_{D_{\text{retr}}}(x_t|c)$) and $p_t(x_t|c) = p^t_D(x_t|c) + p^t_{D_{\text{retr}}}(x_t|c)$.

Figures (7)

  • Figure 1: RAG vs CPR image generation. Images generated using the given prompt for a fixed random seed using different methods. Safe Model: Pre-trained model with no access to the retrievable data store, Retrieval-Score: Image generated using \ref{['eq:retrieval-score']}, Retrieval-Mix-Score: Image generated using \ref{['eq:retrieval-score-complete']}, CPR-KL, CPR-Min, CPR-Alt: Images generated using our algorithms in \ref{['algo:cpr-kl']}\ref{['cpr-min']} and \ref{['algo:cpr-int']}\ref{['cpr-alt']}. Images generated without CPR bear more resemblance to the retrieved image, compared to the CPR generated images, which are different from the retrieved image, while preserving the underlying concept in the prompt (for example the astronaut seems to be on Moon, Big Ben is more textured with different design).
  • Figure 2: (A) We plot the histogram of $\Delta_{\text{max}} = \log \dfrac{p(x|c)}{\operatorname{safe}(x|c)}$ as we vary the contribution of the retrieval-score ($\hat{w_1}$ in \ref{['eq:retrieval-score-complete']}). We use $\hat{w_1}$ as a user tunable parameter which controls the amount of bits the generated images are different from $\operatorname{safe}$. We show that as we reduce $\hat{w_1}$, empirical $k_c$ (max value on the x-axis with non-zero probability) decreases. (B) Comparison to baseline, vyas2023provable, with k=1500 using rejection sampling. Smaller k leads to slow generation which is evident from the distribution.
  • Figure 3: Concept similarity with CPR: In this figure we show the CLIP similarity between CPR generated images and the textual prompt (Syn-Cap) and the retrieved images (Syn-Ret) respectively. We show that while the CPR generated image preserves the concept presented in the textual prompt (their similarity with the caption is high), they do not copy the private retrieved images (their similarity with the retrieved samples is low).
  • Figure 4: (a) Plot of the utility (generation quality) for increasing values of copyright protection, on samples from the MS-COCO dataset. (b) The TIFA score of CPR increases as the size of the retrieval dataset grows. (c) Computational costs of CPR (ours) and CP-Kvyas2023provable compared to the base model.
  • Figure 5:
  • ...and 2 more figures

Theorems & Definitions (12)

  • Proposition 1
  • Proposition 2
  • Definition 1: NAF Definition 2.1 in vyas2023provable
  • Theorem 1
  • Lemma 1
  • Proposition 3
  • Lemma 2
  • proof
  • proof
  • proof
  • ...and 2 more