Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests

TL;DR

The paper tackles the subjective problem of visual interestingness by comparing human judgments with GPT-4o and other LMMs on both single images and image pairs. It introduces a 1,000-image Flickr dataset with multi-source annotations and distills the insights into a learning-to-rank model using CLIP features, demonstrating GPT-4o's superior alignment with humans relative to prior approaches. The work reveals strong agreement in many cases, but also systematic biases and gaps when assessing individual images, underscoring the value and limits of LMM-based annotations for scalable labeling. The findings suggest practical utility in leveraging LMMs for large-scale interestingness studies and knowledge distillation, while guiding future work on demographic effects and broader image domains.

Abstract

Our daily life is highly influenced by what we consume and see. Attracting and holding one's attention -- the definition of (visual) interestingness -- is essential. The rise of Large Multimodal Models (LMMs) trained on large-scale visual and textual data has demonstrated impressive capabilities. We explore these models' potential to understand to what extent the concepts of visual interestingness are captured and examine the alignment between human assessments and GPT-4o's, a leading LMM, predictions through comparative analysis. Our studies reveal partial alignment between humans and GPT-4o. It already captures the concept as best compared to state-of-the-art methods. Hence, this allows for the effective labeling of image pairs according to their (commonly) interestingness, which are used as training data to distill the knowledge into a learning-to-rank model. The insights pave the way for a deeper understanding of human interest.

Figures (5)

• Figure 1: LMMs, such as GPT-4o, encode human-like knowledge and perform well across various tasks. We explore image interestingness, a highly subjective concept, by examining consistent labeling between humans and LMMs and their level of agreement. Pairwise labeling accesses relative measures, which are used to train a rank model to finally assess an image's interestingness.
• Figure 2: Examples of images and corresponding responses from human annotators and GPT-4o. Almost all images are consistently labeled, most of them as interesting.
• Figure 3: Image pairs illustrating instances where humans and GPT-4o agree and disagree. For example, humans and GPT-4o have differing opinions regarding images of insects and birds. At first glance, it may not be immediately evident that the insect image is a metallic sculpture, which could explain why people did not find it as interesting—humans may not give the image the same level of attention as a machine.
• Figure 4: Explanations: The clusters are derived from text embeddings of responses "Why" an image is interesting. Clusters of common interest (a) include "cute", "joyful" moments as well as "uniqueness". A minority of clusters (b) shows disagreement between humans and GPT-4o.
• Figure 5: Image Content Descriptions: Text embeddings of the image descriptions are divided into semantically similar groups using hierarchical clustering (a, b). Most clusters indicate agreement between humans and GPT-4 (c, d), while some indicate disagreement (e). Compare with Tab. \ref{['tab:cluster_agreement_sorted']}.