CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion

TL;DR

This work presents Contrasting Gradient Inversion for Diffusion Models (CGI-DM), a novel method featuring vivid visual representations for digital copyright authentication that surpasses alternative validation techniques in digital copyright authentication.

Abstract

Diffusion Models (DMs) have evolved into advanced image generation tools, especially for few-shot generation where a pretrained model is fine-tuned on a small set of images to capture a specific style or object. Despite their success, concerns exist about potential copyright violations stemming from the use of unauthorized data in this process. In response, we present Contrasting Gradient Inversion for Diffusion Models (CGI-DM), a novel method featuring vivid visual representations for digital copyright authentication. Our approach involves removing partial information of an image and recovering missing details by exploiting conceptual differences between the pretrained and fine-tuned models. We formulate the differences as KL divergence between latent variables of the two models when given the same input image, which can be maximized through Monte Carlo sampling and Projected Gradient Descent (PGD). The similarity between original and recovered images serves as a strong indicator of potential infringements. Extensive experiments on the WikiArt and Dreambooth datasets demonstrate the high accuracy of CGI-DM in digital copyright authentication, surpassing alternative validation techniques. Code implementation is available at https://github.com/Nicholas0228/Revelio.

Figures (10)

• Figure 1: Top: Infringements brought by few-shot generation. A small set of images is used to fine-tuning the pretrained model. The fine-tuned model is then capable of high-quality generation, which, if performed without proper authorization, may lead to infringements. Bottom: Our method CGI-DM for copyright authentication. CGI-DM recovers the missing details from partial representation of an input sample. Then the the similarity between recovered samples and input samples can be used to validate infringements.
• Figure 2: Our framework, CGI-DM, for copyright authentication. We begin with a given image $x_{0}$. ① We first remove part of $x_{0}$, obtaining a partial representation $\overline{x}_{0}$ of the image (refer to \ref{['partial']}). This partial representation is then fed into the optimization loop. ② Within the optimization loop, we leverage the conceptual disparity between the pretrained model $\theta$ and the fine-tuned model $\theta'$ given the partial representation $\overline{x}_{0}$ (refer to \ref{['kl']}) to recover the missing details. Such disparity is formulated as $L_{tar}$. ③ Subsequently, we employ Monte Carlo sampling on time variable $t$ and random noise $\varepsilon_{t}$ to optimize $L_{tar}$ (refer to \ref{['alg']}), getting step-wise gradient for updating the image. Over $\rm{N}$ steps of updating, the optimization loop produces the final recovered image $\overline{x}_{0}^{(N)}$. ④ We authenticate copyright by measuring the similarity between recovered image $\overline{x}_{0}^{(N)}$ with the original image $x_{0}$.
• Figure 3: CGI-DM under different training steps and extraction steps. All other parameters are set the same as those in \ref{['models']}.
• Figure 4: Visualization of different methods for removing partial information and corresponding recovered samples for CGI-DM on Van Gogh's paintings from the WikiArt dataset.
• Figure 5: Comparison of CGI-DM and direct GI under different partial representations. We can find that direct GI does not recover any semantic information of the images.