Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion

TL;DR

The paper tackles the descriptiveness gap in standard image captions by proposing a training-free, three-stage pipeline that (i) generates captions from multiple SoTA models, (ii) ranks them with a new image-text matching metric BLIPScore, and (iii) fuses the top two captions using a Large Language Model. The approach achieves stronger image-text alignment and linguistic richness (notably SPICE) and improves caption diversity on MS-COCO and Flickr30k, supported by subjective human judgments. It demonstrates that combining diverse captioners can overcome the limitations of single-model outputs and provides a practical method for generating richer training data for downstream vision-language tasks. Additionally, the work includes extensive ablations, a subjective study, and an evaluation of finetuning existing models on fused captions, highlighting potential benefits for model generalization and real-world captioning applications.

Abstract

State-of-The-Art (SoTA) image captioning models are often trained on the MicroSoft Common Objects in Context (MS-COCO) dataset, which contains human-annotated captions with an average length of approximately ten tokens. Although effective for general scene understanding, these short captions often fail to capture complex scenes and convey detailed information. Moreover, captioning models tend to exhibit bias towards the ``average'' caption, which captures only the more general aspects, thus overlooking finer details. In this paper, we present a novel approach to generate richer and more informative image captions by combining the captions generated from different SoTA captioning models. Our proposed method requires no additional model training: given an image, it leverages pre-trained models from the literature to generate the initial captions, and then ranks them using a newly introduced image-text-based metric, which we name BLIPScore. Subsequently, the top two captions are fused using a Large Language Model (LLM) to produce the final, more detailed description. Experimental results on the MS-COCO and Flickr30k test sets demonstrate the effectiveness of our approach in terms of caption-image alignment and hallucination reduction according to the ALOHa, CAPTURE, and Polos metrics. A subjective study lends additional support to these results, suggesting that the captions produced by our model are generally perceived as more consistent with human judgment. By combining the strengths of diverse SoTA models, our method enhances the quality and appeal of image captions, bridging the gap between automated systems and the rich and informative nature of human-generated descriptions. This advance enables the generation of more suitable captions for the training of both vision-language and captioning models.

Figures (10)

• Figure 1: Comparison between ground-truth captions (Original) and enriched captions generated by our method (Ours) for sample images from the COCO dataset.
• Figure 2: Overview of the proposed image captioning pipeline.
• Figure 3: Generated captions for two images of the MS-COCO test set.
• Figure 4: Image-text matching scores for the captions generated by using the five considered models on an MS-COCO test image. The captioning models are reported in descending order with respect to the BLIPScore.
• Figure 5: Human-written captions versus generated captions. For this sample image of the MS-COCO test set we report the ground-truth captions, those generated by the five existing models, and the one generated by the proposed model.