WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models

TL;DR

The paper tackles the problem of attributing responsibility for images produced by text-to-image diffusion models by introducing WOUAF, a distributor-oriented fingerprinting method that embeds per-user signals into the decoder via weight modulation. This approach achieves near-perfect attribution accuracy with negligible degradation to image quality and demonstrates robustness against diverse post-processing and deliberate removal attempts, outperforming baselines. It is computationally efficient, requiring only a single forward pass to generate a fingerprinted model, and scalable to billions of users. The work lays groundwork for accountable model distribution and practical safeguards against misuse, with code available for replication and extension to other modalities in future work.

Abstract

The rapid advancement of generative models, facilitating the creation of hyper-realistic images from textual descriptions, has concurrently escalated critical societal concerns such as misinformation. Although providing some mitigation, traditional fingerprinting mechanisms fall short in attributing responsibility for the malicious use of synthetic images. This paper introduces a novel approach to model fingerprinting that assigns responsibility for the generated images, thereby serving as a potential countermeasure to model misuse. Our method modifies generative models based on each user's unique digital fingerprint, imprinting a unique identifier onto the resultant content that can be traced back to the user. This approach, incorporating fine-tuning into Text-to-Image (T2I) tasks using the Stable Diffusion Model, demonstrates near-perfect attribution accuracy with a minimal impact on output quality. Through extensive evaluation, we show that our method outperforms baseline methods with an average improvement of 11\% in handling image post-processes. Our method presents a promising and novel avenue for accountable model distribution and responsible use. Our code is available in \url{https://github.com/kylemin/WOUAF}.

Figures (16)

• Figure 1: Illustration of user attribution based on our method. Please refer to the main text for detailed descriptions.
• Figure 2: Depiction of our method's pipeline and weight modulation: (a) The model fingerprinting procedure encompasses encoding via the mapping network and weight modulation, along with decoding through the fingerprint decoding network. (b) Weight modulation of the decoding network $\mathcal{D}$ to incorporate the fingerprint.
• Figure 3: Qualitative comparison of the original and fingerprinted Stable Diffusion models on MS-COCO lin2014microsoft and LAION aesthetics schuhmann2022laion (Pixel-wise differences$\times$ 5: they are multiplied by a factor of 5 for better view). We can observe that our method maintains high image quality.
• Figure 4: Comparative analysis of weight modulation on decoder $\mathcal{D}$ and diffusion model $\epsilon_{\theta}$ with decoder $\mathcal{D}$. Modulating the diffusion model negatively affects image quality.
• Figure 5: Enhanced Robustness Against Image Post-Processes. For almost all scenarios, WOUAF consistently exceeds the performance of DAG kim2020decentralized and Stable Signature stable_signature.