MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

TL;DR

MIRe addresses the text-dominant bias in multimodal query retrieval by removing fusion during alignment and enabling the textual query to attend to visual embeddings through a query-guided pooling mechanism, while applying late-interaction scoring to fuse modalities. It introduces a response-to-passage pre-training dataset that exposes the model to realistic retrieval scenarios and demonstrates strong zero-shot and fine-tuned performance across four multimodal benchmarks. The approach combines ViT-based visual embeddings with a ColBERTv2-style textual encoder, freezes the text and vision encoders during alignment, and uses a specialized loss and data construction to promote robust cross-modal understanding. Overall, MIRe advances fusion-free modality interaction for multimodal retrieval and provides practical improvements and insights for multimodal search systems.

Abstract

Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe

Figures (8)

• Figure 1: Effect of the text-dominant issue in multimodal query retrieval.
• Figure 2: Overview of the MIRe architecture. This figure illustrates the interaction between the text encoder $\mathcal{R}_T$ and the vision encoder $\mathcal{R}_V$.
• Figure 3: Our data construction process. Starting with visual dialogue datasets, our process involves two steps to convert the dialogue tasks to knowledge retrieval tasks. After preprocessing, we transform responses into a passage format by unifying the response and relevant passages retrieved from Wikipedia.
• Figure 4: Visualization of multimodal query processing, illustrating the alignment between textual and visual modalities.
• Figure 5: Training convergence and retrieval performance. All models were trained for only one epoch under the same settings.