Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samples

Shuhei Yamashita, Daiki Shirafuji, Tatsuhiko Saito

TL;DR

This work tackles the modality gap in vision-language models during multi-modal retrieval by standardizing cross-modal similarity scores using modality-specific statistics estimated from pseudo-positive pairs constructed from unlabeled queries. The method avoids manual labeling or image captioning, yet achieves large gains in cross-modal retrieval, particularly for image queries, across seven pre-trained VLMs on MMQA and WebQA. Key contributions include a formalization of modality-aware similarity normalization, a pseudo data construction framework, and comprehensive analyses showing robustness and effectiveness even when labeled data is unavailable. The results demonstrate that calibrated similarity scoring can bridge cross-modal gaps while preserving rich visual information, offering a practical, labeling-efficient path for real-world multi-modal retrieval systems.

Abstract

Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.

Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samples

TL;DR

This work tackles the modality gap in vision-language models during multi-modal retrieval by standardizing cross-modal similarity scores using modality-specific statistics estimated from pseudo-positive pairs constructed from unlabeled queries. The method avoids manual labeling or image captioning, yet achieves large gains in cross-modal retrieval, particularly for image queries, across seven pre-trained VLMs on MMQA and WebQA. Key contributions include a formalization of modality-aware similarity normalization, a pseudo data construction framework, and comprehensive analyses showing robustness and effectiveness even when labeled data is unavailable. The results demonstrate that calibrated similarity scoring can bridge cross-modal gaps while preserving rich visual information, offering a practical, labeling-efficient path for real-world multi-modal retrieval systems.

Abstract

Advances in vision-language models (VLMs) have enabled effective cross-modality retrieval. However, when both text and images exist in the database, similarity scores would differ in scale by modality. This phenomenon, known as the modality gap, hinders accurate retrieval. Most existing studies address this issue with manually labeled data, e.g., by fine-tuning VLMs on them. In this work, we propose a similarity standardization approach with pseudo data construction. We first compute the mean and variance of the similarity scores between each query and its paired data in text or image modality. Using these modality-specific statistics, we standardize all similarity scores to compare on a common scale across modalities. These statistics are calculated from pseudo pairs, which are constructed by retrieving the text and image candidates with the highest cosine similarity to each query. We evaluate our method across seven VLMs using two multi-modal QA benchmarks (MMQA and WebQA), where each question requires retrieving either text or image data. Our experimental results show that our method significantly improves retrieval performance, achieving average Recall@20 gains of 64% on MMQA and 28% on WebQA when the query and the target data belong to different modalities. Compared to E5-V, which addresses the modality gap through image captioning, we confirm that our method more effectively bridges the modality gap.

Paper Structure

This paper contains 27 sections, 4 equations, 24 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Modal Retrieval
  4. Bridging the Modality Gap
  5. Task Formulation
  6. Proposed Methods
  7. Modality-Specific Similarity Standardization
  8. Pseudo Pair Construction
  9. Experimental Setup
  10. Datasets for Evaluation
  11. Datasets for Pseudo Pair Construction
  12. Metrics
  13. Models
  14. Evaluation Conditions
  15. Results and Discussions
  16. ...and 12 more sections

Figures (24)

  • Figure 1: Conceptual overview of the modality gap. Texts and their corresponding images are projected to distant regions of the embedding space.
  • Figure 2: Overview of our proposed method. The modality gap causes irrelevant text to score higher than relevant images. Our approach addresses this issue by standardizing cosine similarity scores based on modality-specific mean and variance calculated from pseudo data.
  • Figure 3: Examples of positive images for ImageQ in MMQA and WebQA shown in Table \ref{['table: dataset_example']}.
  • Figure 4: MMQA
  • Figure 5: WebQA
  • ...and 19 more figures