Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval

TL;DR

This work tackles the limitation that Multimodal Large Language Models (MLLMs) are often used as embedding extractors without exploiting their reasoning capabilities. It introduces Reasoning Guided Embeddings (RGE), where an MLLM first generates structured rationales conditioned on the input before producing embeddings, effectively coupling reasoning with representation learning. The method jointly optimizes a language modeling objective for rationale generation and a contrastive objective for embedding alignment, while employing on-the-fly rationale generation to avoid information leakage from oracle rationales. On the MMEB benchmark, RGE achieves a 4.9% improvement over non-reasoning baselines at comparable model scales, highlighting reasoning as a practical mechanism to enhance multimodal retrieval and cross-modal representation quality. The approach requires no architectural changes and supports reasoning-aware inference for better, more contextually grounded embeddings.

Abstract

Multimodal embeddings are widely used in downstream tasks such as multimodal retrieval, enabling alignment of interleaved modalities in a shared representation space. While recent studies show that Multimodal Large Language Models (MLLMs) can serve as strong embedding extractors, existing approaches treat embedding extraction as a direct encoding step, overlooking the fact that MLLMs possess the generative capability for reasoning that could be leveraged to enhance representation quality. In this work, we explore how to explicitly incorporate reasoning into the embedding process. To this end, we propose Reasoning Guided Embeddings (RGE), which preserves the generative rationale process of MLLMs and couples it with contrastive training. Our method first enables the model to perform structured rationale generation conditioned on the instruction, and then extracts representations after reasoning has unfolded. This simple design enhances the context-conditional inference signals within the embedding, leading to improved multimodal representation quality. Experiments on the MMEB benchmark show that reasoning-guided conditioning improves multimodal retrieval performance by 4.9% over the non-reasoning baseline, confirming that explicit reasoning can effectively enhance embedding quality.

Figures (5)

• Figure 1: Comparison between our method and non-reasoning baseline. Unlike previous MLLM embedders that directly encode inputs into embeddings without reasoning, our MLLM embedding model (RGE) first performs reasoning over the input before embedding extraction, then uses both the query and its generated rationale to produce a more informative embedding. The illustrated example is a composed image retrieval task (text–image to image) from the CIRR CIRR subset in MMEB jiang2025vlmvec benchmark.
• Figure 2: Overview of our training pipeline. Our training framework jointly optimizes a language modeling loss and a contrastive loss. To mitigate the information leakage problem discussed in \ref{['sec:info-leak']}, the model generates rationales on-the-fly during training, ensuring that the embeddings used for contrastive supervision remain independent of oracle rationales and lead to better representation quality.
• Figure 3: Our dataset curation pipeline. We provide both the query $q$ and its ground-truth target $t$ to an MLLM rationale generator, along with task- and modality-aware prompts designed to produce well-formatted oracle rationales $r_o$. The illustration is selected from the text-image to image grounding task in MMEB.
• Figure 4: Retrieval Experiment of Information Leakage. We conduct diagnostic experiments showing that using the <emb> hidden state from the autoregressive LM loss for contrastive training leads to information leakage. When the model is trained on unpaired queries $q_w$ but with correctly paired oracle rationales $r_o$ and targets $t$, namely $(q_w, r_o)\rightarrow t$, the contrastive loss still converges, indicating that the oracle rationale enables the model to take a shortcut rather than learning true query–target alignment.
• Figure 5: Failure Case Demonstration in the MSCOCO coco Text Retrieval Task. We highlight hallucinated content that does not exist in the falsely retrieved target text in red. We also include the ground-truth captions for the query image and the image corresponding to the falsely retrieved text.