Model Synthesis for Zero-Shot Model Attribution

TL;DR

This work tackles zero-shot model attribution by moving beyond closed-set classifiers to a fingerprint-distance framework trained on a large corpus of synthetic models. It introduces a model-synthesis strategy that yields 5760 shallow, diverse synthetic architectures to emulate real model fingerprints, and trains a ResNet50-based fingerprint extractor using a joint CE and triplet loss, with spectral fingerprints extracted via Discrete Fourier Transform on denoised images. The authors define Frechet Frequency Distance (FFD) to quantify fidelity between real and synthetic fingerprint distributions and demonstrate strong zero-shot generalization across GANs, VAEs, flows, and diffusion models, including Stable Diffusion and DALL-E variants, achieving substantial gains in model identification (>40%) and verification (>15%) on unseen models. The approach also enables tracing LoRA-based variants to their base models, offering a scalable defense against IP infringements while enabling dynamic gallery updates without retraining on new real models.

Abstract

Nowadays, generative models are shaping various fields such as art, design, and human-computer interaction, yet accompanied by challenges related to copyright infringement and content management. In response, existing research seeks to identify the unique fingerprints on the images they generate, which can be leveraged to attribute the generated images to their source models. Existing methods, however, are constrained to identifying models within a static set included in the classifier training, failing to adapt to newly emerged unseen models dynamically. To bridge this gap, we aim to develop a generalized model fingerprint extractor capable of zero-shot attribution, effectively attributes unseen models without exposure during training. Central to our method is a model synthesis technique, which generates numerous synthetic models mimicking the fingerprint patterns of real-world generative models. The design of the synthesis technique is motivated by observations on how the basic generative model's architecture building blocks and parameters influence fingerprint patterns, and it is validated through two designed metrics that examine synthetic models' fidelity and diversity. Our experiments demonstrate that this fingerprint extractor, trained solely on synthetic models, achieves impressive zero-shot generalization on a wide range of real-world generative models, improving model identification and verification accuracy on unseen models by over 40% and 15%, respectively, compared to existing approaches.

Figures (21)

• Figure 1: Illustration of the existing methods (above) and ours (below). Existing methods depend on training with a limited set of real-world models, struggling to distinguish emerging unseen models like Stable Diffusion (SD) and DALL-E series. In contrast, our method trains on numerous synthetic models that replicate a broader range model fingerprints, enabling it to distinguish and attribute unseen models effectively.
• Figure 2: Images (top) produced by generative models and the associated (averaged) Fourier spectrum (bottom). The unique characteristics of the Fourier spectrum can be regarded as the fingerprints of generative models and used for model attribution. The spectrum are obtained from high-pass filtered images processed through a denoiser.
• Figure 3: (a-c) Verification of the importance of the last generative block in producing model fingerprints. In training (a), images generated by two models are used to train a 2-way classifier. In the first testing scenario (b), we test on non-overlap testing samples from the two models. In the second testing scenario (c), we test on samples from two composed models, which are constructed by switching the parameters of the last generative blocks of two models in training. (d-e) The confusion matrices in first and second testing scenarios. Model 1 and Model 2 are abbreviated as M1 and M2, respectively. Composed Model 1 and Composed Model 2 are abbreviated as CM1 and CM2.
• Figure 4: Our proposed model synthesis strategy. The synthetic model receives a real image as input, producing a fingerprinted image that contains the distinct fingerprint characteristics associated with the synthetic model.
• Figure 5: (a) Spectrum pattern representation on a single model. (b) The calculation process of Frechet Frequency Distance.