Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does CLIP perceive art the same way we do?

Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, Nico Wu

TL;DR

This work scrutinizes whether CLIP's vision-language alignment mirrors human art perception by probing semantic content, style, historical context, and artifacts in both human-made and AI-generated paintings. Using two richly annotated datasets and a fixed CLIP encoder, the study combines descriptive-task probing, style classification, and latent-space visualization (UMAP) to quantify alignment and gaps. Key findings show strong semantic grounding but limited sensitivity to stylistic nuance and artistic period, with artifacts and deformations often overlooked unless defect signals are explicitly integrated. The results highlight the need for perceptually grounded supervision and richer metadata to make multimodal systems more faithful evaluators and guides in creative domains. The work informs downstream use in generative guidance and style transfer, signaling practical implications for developing interpretable, art-aware multimodal models.

Abstract

CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it 'see' the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.

Does CLIP perceive art the same way we do?

TL;DR

This work scrutinizes whether CLIP's vision-language alignment mirrors human art perception by probing semantic content, style, historical context, and artifacts in both human-made and AI-generated paintings. Using two richly annotated datasets and a fixed CLIP encoder, the study combines descriptive-task probing, style classification, and latent-space visualization (UMAP) to quantify alignment and gaps. Key findings show strong semantic grounding but limited sensitivity to stylistic nuance and artistic period, with artifacts and deformations often overlooked unless defect signals are explicitly integrated. The results highlight the need for perceptually grounded supervision and richer metadata to make multimodal systems more faithful evaluators and guides in creative domains. The work informs downstream use in generative guidance and style transfer, signaling practical implications for developing interpretable, art-aware multimodal models.

Abstract

CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it 'see' the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.
Paper Structure (14 sections, 7 figures, 5 tables)

This paper contains 14 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology and Data
  4. Experiments on human artworks
  5. Image-description alignment
  6. Style Recognition
  7. Semantic Relationships in the Latent Space
  8. Experiments with AI-generated artworks
  9. Image-prompt similarity
  10. Style Recognition
  11. Comparison with human evaluations
  12. Adherence Analysis
  13. Perception of Artifacts and Deformations
  14. Conclusion

Figures (7)

  • Figure 1: 3D UMAP projection of image embeddings extracted by the CLIP ViT-L/14 model from the National Gallery of Art.
  • Figure 2: Comparison between actual and predicted distribution of styles in NGAD.
  • Figure 3: Examples of misclassifications in style recognition
  • Figure 4: Three-dimensional projection of the textual embeddings of artistic styles and the visual embeddings of correctly classified (green) and misclassified (red) images using UMAP.
  • Figure 5: Visual comparison between a misclassified image (left) and a correctly classified one (right). For each artwork, the actual and predicted artistic styles are shown, along with their respective similarity scores to the image in CLIP's space.
  • ...and 2 more figures