Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets

TL;DR

The paper tackles image captioning with heterogeneous data sources by explicitly separating semantic content from descriptive style. It introduces a style token and retrieval-based keywords to condition a single-objective transformer model, enabling fluent captions that richly describe real-world concepts while leveraging web-scale data without object detectors. Empirical results across COCO, nocaps, CC3M, and other datasets show state-of-the-art performance with strong zero-shot, long-tail, and named-entity capabilities, demonstrating robust cross-domain generalization. This approach offers a scalable path to high-quality captions by harmonizing semantic breadth from web data with the fluency of human-annotated descriptions.

Abstract

This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources, containing both human-annotated and web-collected captions. Large-scale datasets with noisy image-text pairs, indeed, provide a sub-optimal source of supervision because of their low-quality descriptive style, while human-annotated datasets are cleaner but smaller in scale. To get the best of both worlds, we propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component. The proposed model avoids the need of object detectors, is trained with a single objective of prompt language modeling, and can replicate the style of human-collected captions while training on sources with different input styles. Experimentally, the model shows a strong capability of recognizing real-world concepts and producing high-quality captions. Extensive experiments are performed on different image captioning datasets, including CC3M, nocaps, and the competitive COCO dataset, where our model consistently outperforms baselines and state-of-the-art approaches.

Figures (11)

• Figure 1: Sample descriptions generated by our model, in comparison with a Transformer-based captioner trained on COCO. Our approach generates high-quality captions by separating content from style.
• Figure 2: Samples of human-annotated and web-collected (image, caption) pairs and overview of our approach.
• Figure 3: Illustration of the overall structure of our approach, which is composed of an encoder module, a keyword-extraction module, and a decoder module.
• Figure 4: Sample textual keywords extracted on COCO images.
• Figure 5: Comparison of captions generated by VinVL and those generated by our approach on sample images from nocaps.