ProMark: Proactive Diffusion Watermarking for Causal Attribution

TL;DR

ProMark is proposed, a causal attribution technique to attribute a synthetically generated image to its training data concepts like objects, motifs, templates, artists, artists, or styles, and can maintain image quality whilst outperforming correlation-based attribution.

Abstract

Generative AI (GenAI) is transforming creative workflows through the capability to synthesize and manipulate images via high-level prompts. Yet creatives are not well supported to receive recognition or reward for the use of their content in GenAI training. To this end, we propose ProMark, a causal attribution technique to attribute a synthetically generated image to its training data concepts like objects, motifs, templates, artists, or styles. The concept information is proactively embedded into the input training images using imperceptible watermarks, and the diffusion models (unconditional or conditional) are trained to retain the corresponding watermarks in generated images. We show that we can embed as many as $2^{16}$ unique watermarks into the training data, and each training image can contain more than one watermark. ProMark can maintain image quality whilst outperforming correlation-based attribution. Finally, several qualitative examples are presented, providing the confidence that the presence of the watermark conveys a causative relationship between training data and synthetic images.

Figures (13)

• Figure 1: Causative vs. correlation-based matching for concept attribution. ProMark identifies the training data most responsible for a synthetic image ('attribution'). Correlation-based matching doesn't always perform the data attribution properly. We propose ProMark, which is a proactive approach involving adding watermarks to training data and recovering them from the synthetic image to perform attribution in a causative way.
• Figure 2: Overview of ProMark. We show the training and inference procedure for our proposed method. Our training pipeline involves two stages, image encryption and generative model training. We convert the bit-sequences to spatial watermarks ($\boldsymbol{W}$), which are then added to the corresponding concept images ($\boldsymbol{X}$) to make them encrypted ($\boldsymbol{X}_W$). The generative model is then trained with the encrypted images using the LDM supervision. During training, we recover the added watermark using the secret decoder ($\mathcal{D}_S$) and apply the BCE supervision to perform attribution. To sample newly generated images, we use a Gaussian noise and recover the bit-sequences using the secret decoder to attribute them to different concepts. Best viewed in color.
• Figure 3: Example training and newly sampled images of different datasets for the corresponding classes. We observe a similar content in the inference image compared with the training image of the predicted class.
• Figure 4: Visual results of prior embedding-based works. We show the image of the closest matched embedding for each method on ImageNet. We highlight images green for correct attribution, otherwise red. Embedding-based works do not always attribute to the correct concept.
• Figure 5: Ablation experiments: We show the results for ablating multiple parameters of ProMark. (a) Number of concepts, (b) watermark strength, and (c) number of images per concept.