Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang

TL;DR

It is argued that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption, and further proposed Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations.

Abstract

Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.The code will be released when the paper is accepted.

Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning

TL;DR

It is argued that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption, and further proposed Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations.

Abstract

Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.The code will be released when the paper is accepted.
Paper Structure (25 sections, 4 equations, 8 figures, 7 tables)

This paper contains 25 sections, 4 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Supervised Fine-Tuning
  4. Reinforcement Learning
  5. Training-Free
  6. Method
  7. Verifying the Existence of Information Loss
  8. Quantitative Evaluation of the Information Loss in Modality Conversion
  9. Cross-modal Identity Mapping
  10. Expriment
  11. Experimental Settings
  12. Reinforcement Learning on the Base Model
  13. Reinforcement Learning on the SFT Model
  14. Ablation Study
  15. Scalability of the Retrieval Corpus
  16. ...and 10 more sections

Figures (8)

  • Figure 1: Qualitative comparison of retrieved images using different captions. The more detailed the caption, the higher the semantic consistency among the retrieved images. The more accurate the caption, the greater the visual similarity between the retrieved images and the source image.
  • Figure 2: Verifying the effectiveness of our proposed metrics for quantifying information loss via Pearson correlation. For each LVLM, we plot the correlation between average metrics and the logit of breed classification accuracy. Each point corresponds to one bin, with each bin containing 100 samples. The Pearson correlation coefficient is reported for each model. This correlation verifies the effectiveness of our proposed metrics for quantifying information loss during modality conversion.
  • Figure 3: The overall architecture of our proposed Cross-modal Identity Mapping (CIM). We first prompt the LVLM to generate image captions in detail. Based on such captions, CIM then retrieve topK relevant images. In this way, the method analyze the Query-gallery Image Relevance (QIR) and Gallery Representation Consistency (GRC) to infer the information loss between the image and its corresponding caption. Finally, by using CIM as reward function, our method forced LVLMs to generate fine-grained and precise captions.
  • Figure 4: Ablation study of reward components on COCO-LN500 pont2020connecting using Qwen2.5-VL-7B bai2025qwen2.
  • Figure 5: Scalability study of the retrieval corpus on COCO-LN500 pont2020connecting across multiple LVLMs.
  • ...and 3 more figures