Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning

Xiaoxing You, Qiang Huang, Lingyu Li, Chi Zhang, Xiaopeng Liu, Min Zhang, Jun Yu

TL;DR

The paper addresses the challenge of generating journalistically informative news captions that fuse visual content with contextual article knowledge. It introduces MERGE, a multimodal retrieval-augmented generation framework built on an entity-centric multimodal knowledge base (EMKB), a three-stage Hypothesis Caption-guided Multimodal Alignment (HCMA), and Retrieval-driven Multimodal Knowledge Integration (RMKI). Through EMKB, HCMA, and RMKI, MERGE delivers superior caption quality and named entity grounding, achieving state-of-the-art CIDEr and F1 scores on GoodNews and NYTimes800k and strong generalization to Visual News. The work demonstrates that integrating explicit multimodal knowledge with implicit reasoning in a tailored MLLM framework yields robust, domain-adaptive performance for complex, knowledge-intensive vision-language tasks relevant to journalism.

Abstract

News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.

Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning

TL;DR

The paper addresses the challenge of generating journalistically informative news captions that fuse visual content with contextual article knowledge. It introduces MERGE, a multimodal retrieval-augmented generation framework built on an entity-centric multimodal knowledge base (EMKB), a three-stage Hypothesis Caption-guided Multimodal Alignment (HCMA), and Retrieval-driven Multimodal Knowledge Integration (RMKI). Through EMKB, HCMA, and RMKI, MERGE delivers superior caption quality and named entity grounding, achieving state-of-the-art CIDEr and F1 scores on GoodNews and NYTimes800k and strong generalization to Visual News. The work demonstrates that integrating explicit multimodal knowledge with implicit reasoning in a tailored MLLM framework yields robust, domain-adaptive performance for complex, knowledge-intensive vision-language tasks relevant to journalism.

Abstract

News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key challenges: (1) incomplete information coverage, (2) weak cross-modal alignment, and (3) suboptimal visual-entity grounding. To address these issues, we introduce MERGE, the first Multimodal Entity-aware Retrieval-augmented GEneration framework for news image captioning. MERGE constructs an entity-centric multimodal knowledge base (EMKB) that integrates textual, visual, and structured knowledge, enabling enriched background retrieval. It improves cross-modal alignment through a multistage hypothesis-caption strategy and enhances visual-entity matching via dynamic retrieval guided by image content. Extensive experiments on GoodNews and NYTimes800k show that MERGE significantly outperforms state-of-the-art baselines, with CIDEr gains of +6.84 and +1.16 in caption quality, and F1-score improvements of +4.14 and +2.64 in named entity recognition. Notably, MERGE also generalizes well to the unseen Visual News dataset, achieving +20.17 in CIDEr and +6.22 in F1-score, demonstrating strong robustness and domain adaptability.

Paper Structure

This paper contains 63 sections, 7 equations, 11 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods Processing Original Articles
  4. Methods Using Extracted Relevant Contexts
  5. Methods Incorporating MLLMs
  6. The MERGE Framework
  7. Overview
  8. EMKB
  9. Entity Extraction and Image Collection
  10. Background Knowledge Acquisition
  11. EMKB Formulation
  12. HCMA
  13. Stage 1: Hypothesis Caption Generation
  14. Stage 2: Relevant Sentence Selection
  15. Stage 3: Global Summary Generation
  16. ...and 48 more sections

Figures (11)

  • Figure 1: Challenges in news image captioning: (a) Identifying entities absent from the article; (b) Aligning numerical details and visual objects across modalities; (c) Disambiguating entities in images with multiple subjects.
  • Figure 2: Overview of the MERGE framework.
  • Figure 3: Architecture of EMKB, illustrating named entities, their images, background knowledge, and knowledge subgraphs, which support context-rich news image captioning.
  • Figure 4: Case study on GoodNews. Entities correctly identified by MERGE are depicted in blue, while errors are shown in red.
  • Figure 5: Prompt template ($p_k$) used in EMKB to construct knowledge subgraphs.
  • ...and 6 more figures