Unveiling the Mystery of Visual Attributes of Concrete and Abstract Concepts: Variability, Nearest Neighbors, and Challenging Categories

TL;DR

For classifying images of abstract versus concrete concepts, a combination of basic visual features such as color and texture is more effective than features extracted by more complex models like Vision Transformer, but ViTs show better performances in the nearest neighbor analysis, emphasizing the need for a careful selection of visual features when analyzing conceptual variables through modalities other than text.

Abstract

The visual representation of a concept varies significantly depending on its meaning and the context where it occurs; this poses multiple challenges both for vision and multimodal models. Our study focuses on concreteness, a well-researched lexical-semantic variable, using it as a case study to examine the variability in visual representations. We rely on images associated with approximately 1,000 abstract and concrete concepts extracted from two different datasets: Bing and YFCC. Our goals are: (i) evaluate whether visual diversity in the depiction of concepts can reliably distinguish between concrete and abstract concepts; (ii) analyze the variability of visual features across multiple images of the same concept through a nearest neighbor analysis; and (iii) identify challenging factors contributing to this variability by categorizing and annotating images. Our findings indicate that for classifying images of abstract versus concrete concepts, a combination of basic visual features such as color and texture is more effective than features extracted by more complex models like Vision Transformer (ViT). However, ViTs show better performances in the nearest neighbor analysis, emphasizing the need for a careful selection of visual features when analyzing conceptual variables through modalities other than text.

Figures (7)

• Figure 1: Images of concrete and abstract concepts with varying concreteness ratings on a scale from $1$ (clearly abstract) to $5$ (clearly concrete), and two plausible visual representations each. The examples are extracted from the Bing dataset described in Section \ref{['subsec:Images']}.
• Figure 2: Weighted F1-scores for different features and different dataset sizes for Bing and YFCC using RF.
• Figure 3: Class-wise F1-scores of abstract and concrete concepts for RF, across features and dataset sizes.
• Figure 4: Five most frequent reasons (top row) of visual diversity among images associated with the same concept (indicated by the bold font in the example list below each image).
• Figure 5: Weighted F1-scores (overall and by class) for different features and different dataset sizes for Bing and YFCC using Logistic Regression.