Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models

Jingwei Yi, Junhao Yin, Ju Xu, Peng Bao, Yongliang Wang, Wei Fan, Hao Wang

TL;DR

This work formalizes Contextual Image Reference for vision–language systems, addressing the gap where VLMs fail to strategically reference relevant images from retrieved documents during conversations. It introduces ImageRef-VL, an instruction-finetuned, end-to-end framework built on InternVL2, together with CIR-Interleave/CIR-Caption datasets and a CIR-Test evaluation protocol. The approach yields substantial gains over open-source baselines (around 88% in key metrics) while remaining more computationally efficient than three-stage pipelines, demonstrating strong potential for visually-aware multimodal conversational AI. The study provides a practical pathway for integrating contextual imagery into RAG-style dialogues and outlines directions for further improvements, such as dynamic image manipulation and RL-based refinement.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at https://github.com/bytedance/ImageRef-VL.

ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models

TL;DR

This work formalizes Contextual Image Reference for vision–language systems, addressing the gap where VLMs fail to strategically reference relevant images from retrieved documents during conversations. It introduces ImageRef-VL, an instruction-finetuned, end-to-end framework built on InternVL2, together with CIR-Interleave/CIR-Caption datasets and a CIR-Test evaluation protocol. The approach yields substantial gains over open-source baselines (around 88% in key metrics) while remaining more computationally efficient than three-stage pipelines, demonstrating strong potential for visually-aware multimodal conversational AI. The study provides a practical pathway for integrating contextual imagery into RAG-style dialogues and outlines directions for further improvements, such as dynamic image manipulation and RL-based refinement.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at https://github.com/bytedance/ImageRef-VL.
Paper Structure (18 sections, 3 equations, 5 figures, 2 tables)

This paper contains 18 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision Language Models
  4. Retrieval-Augmented Generation
  5. LLM Citation Generation
  6. Problem Definition.
  7. Method
  8. Training Data Construction
  9. Model Training
  10. Evaluation Settings.
  11. Automated Evaluation
  12. Human Evaluation
  13. Experimental Results
  14. Experimental Settings
  15. Performance Comparison
  16. ...and 3 more sections

Figures (5)

  • Figure 1: An example of contextual image reference, where referencing the images of a Brachiosaurus can largely enhance user comprehension and engagement.
  • Figure 2: The training strategy of the proposed IMI-VL model. Stage 1: Training dataset construction involves generating textual responses and image descriptions through a language model and a vision-language model. These are combined into interleaved responses using image contexts and captions. Stage 2: Supervised fine-tuning refines the model with a vision encoder, adapter, and language model, optimizing through generative loss.
  • Figure 3: Human evaluation score distribution of four methods.
  • Figure 4: Impact of the hyper-parameters on our ImageRef-VL-8B and ImageRef-VL-26B.
  • Figure 5: Computational cost comparison between our ImageRef-VL and three-stage methods.