Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning

TL;DR

This work introduces CompreCap, a human-annotated benchmark that encodes images as directed scene graphs—linking objects with bound attributes and directional relations—to enable structured evaluation of detailed captions from Large Vision-Language Models. By decomposing captions into object, attribute, and relation components and employing LLM-based scoring, the approach yields a unified metric that aligns closely with human judgments and outperforms traditional caption metrics for long, dense captions. The dataset comprises 560 images with high object coverage and detailed CompreQA tasks focusing on tiny objects, plus a robust evaluation pipeline validated across 10 LVLMs and human performance. The results demonstrate CompreCap’s practical value for diagnosing model capabilities in comprehensive image understanding and guiding future LVLM development for text-rich visual content.

Abstract

Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs.

Figures (18)

• Figure 1: Data sample of CompreCap dataset. Objects in the image are manually annotated with segmentation maps, semantic labels, their bound attribute descriptions and the relations with other objects, composing a directed scene graph structure. The global scene description is organized by these annotations.
• Figure 2: In the absence of the annotation of the directed scene graph, the extraction of isolated terms of objects, attributes, and relations disrupts the structured scene graph information, resulting in an inability to distinguish the quality of generated captions. We construct the CompreCap which is manually annotated with the attributes bound to objects and the directional relationships between objects. Based on the high-quality CompreCap dataset, we decompose the generated captions into a hierarchical structure and align them with object, attribute, and relation level annotations, enabling a more solid assessment of detailed captions.
• Figure 3: We utilize the prompt to guide LLM to analyze whether the sub-captions contain a similar concept with the given phrase and provide a score from 0 to 5.
• Figure 4: Illustration of a comprehensive caption evaluation example. The words or sub-captions corresponding to the human annotated object, attribute, and relation are in green, underlined in yellow, and highlighted in purple, respectively. GPT-4V fails to identify objects such as the 'fence' in the background compared to LLaVA-Next-34B liu2024llava. Additionally, the attribute and relationship description from LLaVA-Next-34B are more consistent with human annotations.
• Figure 5: The human performance exceeds all LVLMs with our evaluation method. Our annotations provide more precise references when assessing detailed captions.