Fingerprints of Super Resolution Networks

TL;DR

SISR networks with a high upscaling factor or trained using adversarial loss leave highly distinctive fingerprints, and that under certain conditions, some SISR network hyperparameters can be reverse-engineered from these fingerprints.

Abstract

Several recent studies have demonstrated that deep-learning based image generation models, such as GANs, can be uniquely identified, and possibly even reverse-engineered, by the fingerprints they leave on their output images. We extend this research to single image super-resolution (SISR) networks. Compared to previously studied models, SISR networks are a uniquely challenging class of image generation model from which to extract and analyze fingerprints, as they can often generate images that closely match the corresponding ground truth and thus likely leave little flexibility to embed signatures. We take SISR models as examples to investigate if the findings from the previous work on fingerprints of GAN-based networks are valid for general image generation models. We show that SISR networks with a high upscaling factor or trained using adversarial loss leave highly distinctive fingerprints, and that under certain conditions, some SISR network hyperparameters can be reverse-engineered from these fingerprints.

Figures (7)

• Figure 1: An overview of our experimental setup. 1,000 low-resolution images are fed through each of 205 SISR models to produce a dataset of 205,000 super-resolved images. We then train our model parsing and attribution classifiers on this dataset of super-resolved images.
• Figure 2: A small image patch super-resolved by a sample of the SISR models in our dataset. Abbreviations: REGAN: Real-ESRGAN, Sw-$L_1$: L1-optimized SwinIR, Sw-Adv: adversarially-optimized SwinIR, E.Net: EnhanceNet, E.GAN: ESRGAN. Best viewed zoomed in.
• Figure 3: T-SNE visualizations of super-resolved image feature embeddings, grouped by scale and loss. Each point represents an image from the test set, colored according to the SISR model that generated it. Accuracies for each group (for this classifier) are in the lower left.
• Figure 4: Distinguishing between models which differ only by seed. The table shows the accuracy (%) our custom model attribution classifier vs the PRNU-based classifier from marra2017source. One of the 30 seed triplets, the (4X, $L_1$, NLSN) triplet, had a relatively low distinction accuracy of 87.3%. So the table is computed with NLSN models omitted.
• Figure 5: T-SNE Feature embeddings for images generated by our pretrained SISR models. Left: embeddings as encoded by one of the pretrained model attribution classifiers (attribution accuracy in lower-left). Right: embeddings by one of the custom model attribution classifiers, which was not trained on these models.