Synthetic images aid the recognition of human-made art forgeries

TL;DR

This work examines the potential of incorporating synthetic artworks into training datasets to enhance the performance of forgery detection and finds that the inclusion of synthetic forgeries in the training enables the detection of AI-generated forgeries, especially if created using a similar generator.

Abstract

Previous research has shown that Artificial Intelligence is capable of distinguishing between authentic paintings by a given artist and human-made forgeries with remarkable accuracy, provided sufficient training. However, with the limited amount of existing known forgeries, augmentation methods for forgery detection are highly desirable. In this work, we examine the potential of incorporating synthetic artworks into training datasets to enhance the performance of forgery detection. Our investigation focuses on paintings by Vincent van Gogh, for which we release the first dataset specialized for forgery detection. To reinforce our results, we conduct the same analyses on the artists Amedeo Modigliani and Raphael. We train a classifier to distinguish original artworks from forgeries. For this, we use human-made forgeries and imitations in the style of well-known artists and augment our training sets with images in a similar style generated by Stable Diffusion and StyleGAN. We find that the additional synthetic forgeries consistently improve the detection of human-made forgeries. In addition, we find that, in line with previous research, the inclusion of synthetic forgeries in the training also enables the detection of AI-generated forgeries, especially if created using a similar generator.

Figures (9)

• Figure 1: Illustration of real (top row) and synthetic (bottom row) van Gogh images. "Self Portrait with a Straw Hat", Vincent van Gogh (1887) Bailey_2021 (square-cropped, top left), "Self-portrait with a Bandaged Ear and Pipe", sold by Otto Wacker, previously attributed to van Gogh Harvard_2019 (top right), fine-tuned GAN generated image in the style of van Gogh (bottom left), and Stable Diffusion generated image in style of van Gogh (bottom right).
• Figure 2: Composition of the training and testing sets for the different experiments. Each box in the training represents a training configuration. The configuration names on the bottom row are used throughout the following sections. Green sub-boxes indicate the original set, red indicates the contrast set.
• Figure 3: Accuracies of different models for originals and forgeries. Based on the results presented in Tables \ref{['table2']} and \ref{['table3']} with the composition of the underlying van Gogh data set as detailed in Table \ref{['table1']} and visualised in Fig. \ref{['fig2']}. The horizontal dotted line shows the baseline without synthetic images in the training data. Similar results for the artists Modigliani and Raphael can be found in \ref{['S1_Appendix']}.
• Figure 4: Accuracies of different models for synthetic data. Based on the results shown in Table \ref{['table4']} with the composition of the underlying van Gogh data set as detailed in Table \ref{['table1']} and visualised in Fig. \ref{['fig2']}.
• Figure 5: Representative $256\times 256$ images. Categories used are 'history and genre paintings' (top left), 'landscapes' (top right), 'figurative and allegorical' (bottom left), and 'still and flower paintings' (bottom right).