Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

TL;DR

The mechanisms underlying VLMs'ability to predict artistic style are characterized and the extent to which they align with the criteria art historians use to reason about artistic style is assessed.

Abstract

VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detection. This includes increasingly strong capabilities in the domain of art, from analyzing artwork to generation of art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature and 90% of concepts used to predict style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.

Figures (12)

• Figure 1: A motivating example. Top: The VLM classifies the image as Renaissance, however, image concepts offer little explanatory insight---they display visually similar images but do not reveal why the model made this prediction. Relevant tokens obtained via logit lens are similarly non-descriptive. Bottom: Our method extracts interpretable, patch-level concepts from the image, assigning each a label that captures both content and form.
• Figure 2: Overview of the concept decomposition pipeline. In the training stage, we (1) split training images into 4x4 patches, (2) extract their VLM latent representations and decompose them to obtain patch-level concept activations, and (3) generate text labels describing the top activating images of each concept. At test time, given an image, we identify its corresponding image-level concept and map it to the patch-level concepts that are most strongly represented within it and display the top results.
• Figure 3: Overview of VLM performance on zero-shot art style classification (full image).
• Figure 4: Accuracy of a linear probe trained to predict model output style from its concept activations ($\bullet$ raw activations; $\blacktriangle$ binarized activations).
• Figure 5: The space of concepts visualized in 2D via t-SNE. Concepts more similar to each other in high-dimension are closer. The size of each concept indicates how frequently the concept is activated across the dataset. Colors indicate style-specific concepts--instances where $>70\%$ of the activated images correspond to 1 (or if not, 2) styles. These styles either correspond to the model's prediction (rows 1, 3) or the ground truth style (rows 2, 4).