Is Your Image a Good Storyteller?

TL;DR

The paper introduces Image Semantic Assessment (ISA) to quantify semantic complexity in images, addressing a gap in prior work that focused on quality or entity-level metrics. It builds the first ISA dataset (2,946 images) with Entity and Semantic Scores and proposes VLISA, a vision-language framework that uses LVLMs (e.g., GPT-4o) to extract semantic text from images and a discriminator to predict scores, with a Chain-of-Thought variant enhancing semantic scoring. Experiments show language-guided features substantially improve semantic predictions, especially when using Chain-of-Thought descriptions, and the dataset enables automatic discovery and evaluation of richly semantical, storytelling images across cultures. This work has practical implications for selecting semantically rich visuals and guiding generation and cognitive-assessment applications, while providing a robust, open baseline for future research.

Abstract

Quantifying image complexity at the entity level is straightforward, but the assessment of semantic complexity has been largely overlooked. In fact, there are differences in semantic complexity across images. Images with richer semantics can tell vivid and engaging stories and offer a wide range of application scenarios. For example, the Cookie Theft picture is such a kind of image and is widely used to assess human language and cognitive abilities due to its higher semantic complexity. Additionally, semantically rich images can benefit the development of vision models, as images with limited semantics are becoming less challenging for them. However, such images are scarce, highlighting the need for a greater number of them. For instance, there is a need for more images like Cookie Theft to cater to people from different cultural backgrounds and eras. Assessing semantic complexity requires human experts and empirical evidence. Automatic evaluation of how semantically rich an image will be the first step of mining or generating more images with rich semantics, and benefit human cognitive assessment, Artificial Intelligence, and various other applications. In response, we propose the Image Semantic Assessment (ISA) task to address this problem. We introduce the first ISA dataset and a novel method that leverages language to solve this vision problem. Experiments on our dataset demonstrate the effectiveness of our approach.

Figures (6)

• Figure 1: The Cookie Theft pictures. (a) is the original version and (b) is an updated version.
• Figure 2: Samples from the proposed ISA dataset. ES and SS stand for Entity Score and Semantic Score respectively.
• Figure 3: Annotation criteria of Entity Score. The referenced Cookie Theft refers to the updated version.
• Figure 4: Annotation criteria of Semantic Score.
• Figure 5: Pipeline of our proposed VLISA method.