Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Vision: Contextually Enriched Image Captioning with Multi-Modal Retrieva

Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Dao Sy Duy Minh, Nguyen Hoang Minh Ngoc, Huynh Trung Kiet

TL;DR

The paper tackles the gap where image captions fail to convey contextual event information. It introduces a retrieval-augmented multimodal pipeline that fuses visual analysis with external knowledge through semantic retrieval (BEiT-3, SigLIP2), geometric verification (ORB, SIFT), article-context extraction, and QLoRA-finetuned DeepSeek-Qwen3 to enrich base captions produced by InstructBLIP. Key contributions include demonstrated improvements on the OpenEvents v1 benchmark in both retrieval and captioning quality, highlighting the value of integrating cross-modal retrieval with context-aware language modeling for journalism and digital archives. The work shows potential for richer, more informative captions that connect visual content to broader real-world narratives, enabling deeper visual-textual understanding in practical applications.

Abstract

Real-world image captions often lack contextual depth, omitting crucial details such as event background, temporal cues, outcomes, and named entities that are not visually discernible. This gap limits the effectiveness of image understanding in domains like journalism, education, and digital archives, where richer, more informative descriptions are essential. To address this, we propose a multimodal pipeline that augments visual input with external textual knowledge. Our system retrieves semantically similar images using BEIT-3 (Flickr30k-384 and COCO-384) and SigLIP So-384, reranks them using ORB and SIFT for geometric alignment, and extracts contextual information from related articles via semantic search. A fine-tuned Qwen3 model with QLoRA then integrates this context with base captions generated by Instruct BLIP (Vicuna-7B) to produce event-enriched, context-aware descriptions. Evaluated on the OpenEvents v1 dataset, our approach generates significantly more informative captions compared to traditional methods, showing strong potential for real-world applications requiring deeper visual-textual understanding

Beyond Vision: Contextually Enriched Image Captioning with Multi-Modal Retrieva

TL;DR

The paper tackles the gap where image captions fail to convey contextual event information. It introduces a retrieval-augmented multimodal pipeline that fuses visual analysis with external knowledge through semantic retrieval (BEiT-3, SigLIP2), geometric verification (ORB, SIFT), article-context extraction, and QLoRA-finetuned DeepSeek-Qwen3 to enrich base captions produced by InstructBLIP. Key contributions include demonstrated improvements on the OpenEvents v1 benchmark in both retrieval and captioning quality, highlighting the value of integrating cross-modal retrieval with context-aware language modeling for journalism and digital archives. The work shows potential for richer, more informative captions that connect visual content to broader real-world narratives, enabling deeper visual-textual understanding in practical applications.

Abstract

Real-world image captions often lack contextual depth, omitting crucial details such as event background, temporal cues, outcomes, and named entities that are not visually discernible. This gap limits the effectiveness of image understanding in domains like journalism, education, and digital archives, where richer, more informative descriptions are essential. To address this, we propose a multimodal pipeline that augments visual input with external textual knowledge. Our system retrieves semantically similar images using BEIT-3 (Flickr30k-384 and COCO-384) and SigLIP So-384, reranks them using ORB and SIFT for geometric alignment, and extracts contextual information from related articles via semantic search. A fine-tuned Qwen3 model with QLoRA then integrates this context with base captions generated by Instruct BLIP (Vicuna-7B) to produce event-enriched, context-aware descriptions. Evaluated on the OpenEvents v1 dataset, our approach generates significantly more informative captions compared to traditional methods, showing strong potential for real-world applications requiring deeper visual-textual understanding
Paper Structure (18 sections, 6 equations, 1 figure, 3 tables)

This paper contains 18 sections, 6 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. OpenEvents v1
  4. BEiT-3 and SigLIP2
  5. BEiT-3
  6. SigLIP 2
  7. ORB and SIFT
  8. InstructBLIP (Vicuna‑7B) (Salesforce/instructblip‑vicuna‑7b)
  9. Context Chunking and Embedding using all-MiniLM-l12-v2
  10. DeepSeek-R1-0528-Qwen3-8B
  11. Methodology
  12. Image Retrieval and Ensemble Models
  13. Feature Matching & Visual Verification
  14. Base Caption Generation
  15. QLoRA-based Fine-tuning of DeepSeek Qwen3 for Context-Enriched Image Captioning
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Training loss comparison: semantic chunk selection (rank 64) vs. full article input (rank 512).