Exploiting Pseudo Image Captions for Multimodal Summarization

TL;DR

This work tackles multimodal summarization when explicit visual supervision is scarce by introducing pseudo image captions that connect images with text. The core method, SITA, uses a coarse-to-fine image-text alignment to generate high-quality pseudo captions guided by reference captions retrieved from golden summaries, enabling improved text generation and image selection. By training a two-pass ITA with a bipartite matching step and integrating pseudo captions into a dual-attention text summarizer, SITA achieves state-of-the-art results on the MSMO dataset, demonstrating improvements in ROUGE metrics and cross-modal relevance and integrity. The approach offers a practical way to leverage pseudo-captions to bridge the cross-modal semantic gap, with significant implications for multimedia content understanding and retrieval in real-world web-scale data.

Abstract

Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives. In this paper, we study this problem from the perspective of Mutual Information (MI) optimization. It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist. Guided by a more general lower bound form for optimization, we propose a contrastive learning strategy regulated by progressively refined cross-modal similarity, to more accurately optimize MI between an image/text anchor and its negative texts/images instead of improperly minimizing it. Our method performs competitively on four downstream cross-modal tasks and systematically balances the beneficial and harmful effects of (partial) false negative samples under theoretical guidance.

Figures (5)

• Figure 1: Overview of text summarization and MSMO. Compared with text summarization models, existing MSMO methods usually use an extra image encoder to project images into intermediate representations. They identify the salient image by cross-modal attention, which could be inaccurate due to the lack of golden images for training. We explicitly transform an image into a concrete caption by image-text alignment, capturing visual knowledge better and making text summarization and image selection more effective yet simpler.
• Figure 2: Coarse-to-fine Image-Text alignment. The left part (figure a) shows the overview of the whole image-text alignment mechanism. Reference captions are first retrieved from golden summaries based on a cross-modal retrieval model. We then train an image-text alignment model with reference captions as supervision signals, identifying a relevant sentence as a pseudo caption for each image. The right part (figure b) demonstrates how our two-pass coarse-to-fine alignment model works internally.
• Figure 3: ROUGE-1 and ROUGE-L scores of simple summaries generated by simply concatenating pseudo captions (orange) or golden captions (blue) of a document's first $k$ images. The scores are calculated by matching them against the reference summaries. The horizontal red (dashed) lines represent the text summaries generated by SITA. ROUGE-2 is similar to Rouge-1, which is not shown for better visualization.
• Figure 4: ROUGE-2 of simple summaries generated by simply concatenating pseudo captions (red) or golden captions (blue) of a document's first $k$ images. The scores are calculated by matching them against the reference summaries. The horizontal red dashed lines represent the text summaries generated by our SITA model.
• Figure 5: An example of multimodal summarization generation. The text summary in the last column is generated by the single-modal text summarization model BertSum liu2019text. The pictorial summary in the second last column is generated by MOF zhu2020multimodal re-implemented by ourselves. The orange text in the pseudo and gold image captions corresponds to the semantically important entities in red boxes of images.