How Deep is Your Art: An Experimental Study on the Limits of Artistic Understanding in a Single-Task, Single-Modality Neural Network

TL;DR

Experimental results supported the two hypotheses showing that the DCNN model ignores Non-Exhibited Properties and uses only Exhibited Properties for artwork classification, which points to current DCNN limitations, which should be addressed by future DNN models.

Abstract

Computational modeling of artwork meaning is complex and difficult. This is because art interpretation is multidimensional and highly subjective. This paper experimentally investigated the degree to which a state-of-the-art Deep Convolutional Neural Network (DCNN), a popular Machine Learning approach, can correctly distinguish modern conceptual art work into the galleries devised by art curators. Two hypotheses were proposed to state that the DCNN model uses Exhibited Properties for classification, like shape and color, but not Non-Exhibited Properties, such as historical context and artist intention. The two hypotheses were experimentally validated using a methodology designed for this purpose. VGG-11 DCNN pre-trained on ImageNet dataset and discriminatively fine-tuned was trained on handcrafted datasets designed from real-world conceptual photography galleries. Experimental results supported the two hypotheses showing that the DCNN model ignores Non-Exhibited Properties and uses only Exhibited Properties for artwork classification. This work points to current DCNN limitations, which should be addressed by future DNN models.

Figures (15)

• Figure 1: Art analysis according to Levinson’s definition of the elements of arts. (a) Art consists of Exhibited Properties (EXPs), illustrated in a darker color, and None-Exhibited Properties (NEXP), depicted in a lighter color. (b) NEXPs are accessed by relating EXPs to the historical discourse of art. (c) The difficulty in art interpretation is shown as a spectrum with an example for each end, top: "Fountain" by Marcel Duchamp, and bottom: "The Accident" by William Geets.
• Figure 2: Automated art understanding. NEXPs are more critical for an automated activity as more artistic aspects must be considered (i.e. style recognition to gallery recognition).
• Figure 3: Methodology summary. (a) The gap ($\Delta$) between human analysis and DCNN classification is due to limitations of DCNN model. (b) Dataset design method. (c) The methodology for validating Hypothesis I and Hypothesis II.
• Figure 4: Dataset summary. The designed datasets S1, S2, S3, SF1-4, S4 and G1 pertain to a broad difficulty spectrum according to their EXPs and NEXPs dissimilarities and similarities within or between galleries.
• Figure 5: Results for Dataset S1. (a) PCA plots for the training and test data of subsets difficult, average, and easy. (b) Box and whisker plots of the overall accuracies (ACC) of the DCNN classification of the four subsets. Subsets difficult and average have outliers of 20% and 100%.