Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Explainability of Vision-Language Models in Art History

Stefanie Schneider

TL;DR

This paper examines how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts, combining zero-shot localization experiments with human interpretability studies.

Abstract

Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.

On the Explainability of Vision-Language Models in Art History

TL;DR

This paper examines how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts, combining zero-shot localization experiments with human interpretability studies.

Abstract

Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.
Paper Structure (12 sections, 1 equation, 11 figures, 2 tables)

This paper contains 12 sections, 1 equation, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method Selection
  3. Case Study 1
  4. Pre-processing Steps
  5. Data
  6. Results
  7. Case Study 2
  8. Experimental Design
  9. Participants
  10. Results
  11. Discussion
  12. Conclusions

Figures (11)

  • Figure 1: The saliency map highlights, in red, the image regions most strongly associated with the concept of the "snake" in Franz von Stuck's Adam and Eve (c. 1920).
  • Figure 2: For Ercole de' Roberti's The Wife of Hasdrubal and Her Children (c. 1490/1493), the ground-truth bounding boxes for the concept "nude" are shown in green (a). The bounding boxes derived from the class-conditional saliency map are displayed at progressively increasing threshold levels $\tau \in \{ 0.20, 0.30, 0.40, 0.50, 0.60 \}$.
  • Figure 3: Seven artworks were selected for the online study, each paired with two target classes: Petrus Christus, A Goldsmith in his Shop (1449; a) with "convex mirror" and "girdle"; Franz von Stuck, Adam and Eve (c. 1920; b) with "arm outstretched" and "snake"; Antonello da Messina, Calvary (1475; c) with "John" and "thief"; Claude Monet, Japanese Footbridge (1899; d) with "bridge" and "flower"; Jean-Auguste-Dominique Ingres, Oedipus and the Sphinx (1808; e) with "left foot" and "Sphinx"; Sandro Botticelli, The Lamentation (c. 1490; f) with "sword" and "Virgin Mary"; Bartholomeus van der Helst, The Musician (1662; g) with "lustful" and "sheet music."
  • Figure 4: Evaluation results are shown as divergent stacked bar charts, comparing seven visual explainability methods for each image and class. Colors range from red ("least accurate") to blue ("most accurate"). Kendall's $W$ is reported to assess inter-rater reliability.
  • Figure A.1: Ground-truth annotations and saliency maps for Franz von Stuck's Adam and Eve (c. 1920), shown for the classes "arm outstretched" (top row) and "snake" (bottom row).
  • ...and 6 more figures