Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking and Improving Detail Image Caption

Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, Haoyuan Guo

TL;DR

This paper tackles the misalignment between state-of-the-art LVLMs and reliable evaluation in detail image captioning. It introduces DetailCaps, a higher-quality, expert-annotated benchmark, and CAPTURE, an element-centric metric that leverages a T5-based parser, stop-word filtering, and a three-stage matching process to align closely with human judgments. It further presents a five-stage detail caption construction pipeline that synthesizes high-quality captions using only the LVLM and open-source tools, with a self-looping option to further boost data quality. Experimental results show CAPTURE achieves the strongest consistency with expert judgments, while the synthesized data pipeline improves LVLMs’ detail captioning and broader understanding across multiple benchmarks, supporting practical benefits for both evaluation and model training.

Abstract

Image captioning has long been regarded as a fundamental task in visual understanding. Recently, however, few large vision-language model (LVLM) research discusses model's image captioning performance because of the outdated short-caption benchmarks and unreliable evaluation metrics. In this work, we propose to benchmark detail image caption task by curating high-quality evaluation datasets annotated by human experts, GPT-4V and Gemini-1.5-Pro. We also design a more reliable caption evaluation metric called CAPTURE (CAPtion evaluation by exTracting and coUpling coRE information). CAPTURE extracts visual elements, e.g., objects, attributes and relations from captions, and then matches these elements through three stages, achieving the highest consistency with expert judgements over other rule-based or model-based caption metrics. The proposed benchmark and metric provide reliable evaluation for LVLM's detailed image captioning ability. Guided by this evaluation, we further explore to unleash LVLM's detail caption capabilities by synthesizing high-quality data through a five-stage data construction pipeline. Our pipeline only uses a given LVLM itself and other open-source tools, without any human or GPT-4V annotation in the loop. Experiments show that the proposed data construction strategy significantly improves model-generated detail caption data quality for LVLMs with leading performance, and the data quality can be further improved in a self-looping paradigm. All code and dataset will be publicly available at https://github.com/foundation-multimodal-models/CAPTURE.

Benchmarking and Improving Detail Image Caption

TL;DR

This paper tackles the misalignment between state-of-the-art LVLMs and reliable evaluation in detail image captioning. It introduces DetailCaps, a higher-quality, expert-annotated benchmark, and CAPTURE, an element-centric metric that leverages a T5-based parser, stop-word filtering, and a three-stage matching process to align closely with human judgments. It further presents a five-stage detail caption construction pipeline that synthesizes high-quality captions using only the LVLM and open-source tools, with a self-looping option to further boost data quality. Experimental results show CAPTURE achieves the strongest consistency with expert judgments, while the synthesized data pipeline improves LVLMs’ detail captioning and broader understanding across multiple benchmarks, supporting practical benefits for both evaluation and model training.

Abstract

Image captioning has long been regarded as a fundamental task in visual understanding. Recently, however, few large vision-language model (LVLM) research discusses model's image captioning performance because of the outdated short-caption benchmarks and unreliable evaluation metrics. In this work, we propose to benchmark detail image caption task by curating high-quality evaluation datasets annotated by human experts, GPT-4V and Gemini-1.5-Pro. We also design a more reliable caption evaluation metric called CAPTURE (CAPtion evaluation by exTracting and coUpling coRE information). CAPTURE extracts visual elements, e.g., objects, attributes and relations from captions, and then matches these elements through three stages, achieving the highest consistency with expert judgements over other rule-based or model-based caption metrics. The proposed benchmark and metric provide reliable evaluation for LVLM's detailed image captioning ability. Guided by this evaluation, we further explore to unleash LVLM's detail caption capabilities by synthesizing high-quality data through a five-stage data construction pipeline. Our pipeline only uses a given LVLM itself and other open-source tools, without any human or GPT-4V annotation in the loop. Experiments show that the proposed data construction strategy significantly improves model-generated detail caption data quality for LVLMs with leading performance, and the data quality can be further improved in a self-looping paradigm. All code and dataset will be publicly available at https://github.com/foundation-multimodal-models/CAPTURE.
Paper Structure (57 sections, 6 equations, 5 figures, 7 tables)

This paper contains 57 sections, 6 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image caption evaluation.
  4. Detail caption data construction.
  5. Benchmarking Detail Image Caption
  6. Detail Caption Evaluation Datasets
  7. CAPTURE Metric
  8. Visual elements extraction.
  9. Stop words filtering.
  10. Visual elements matching.
  11. Improving Detail Image Caption
  12. Detail Caption Construction
  13. Stage I: Overall caption generation.
  14. Stage II: Salient visual elements detection.
  15. Stage III: Local caption generation.
  16. ...and 42 more sections

Figures (5)

  • Figure 1: An illustration of the proposed detail caption evaluation metric CAPTURE and caption data quality improvement pipeline.
  • Figure 2: An illustration of the proposed detail caption evaluation metric CAPTURE. The crossed text indicates objects discarded by the stop words filtering module.
  • Figure 3: Prompts used to construct the GPT4-Evaluation Score of DetailCaps-4870 dataset. We prompt GPT4 to generate the relative ranking and the absolute score of candidate captions, through three in context learning cases written by human.
  • Figure 4: Prompts used to generate detail caption by GPT-4V and Gemini-Pro-1.5.
  • Figure 5: Comparison of the original LVLM-generated caption and the synthesized caption after detail caption construction. The red annotations represent description errors, and the green annotations in the synthesized captions represent the correct descriptions compared to the LVLM-generated ones.