Reasoning-Augmented Representations for Multimodal Retrieval

TL;DR

The paper tackles brittleness in universal multimodal retrieval by externalizing latent reasoning through a strong Vision–Language Model to create semantically dense text representations for both corpus entries and queries. This reasoning-augmented data turns reasoning-then-retrieve into explicit semantic matching, allowing standard retrievers to focus on compression. Training on these enhanced representations yields consistent gains across the M-BEIR benchmarks, with query enhancement driving improvements in compositional and modification tasks and corpus enhancement boosting knowledge-intensive retrieval. Inference-time enhancements alone are insufficient; aligning the retriever to the dense semantic distribution is essential. The approach offers a practical, data-centric pathway to more robust multimodal retrieval without major architectural changes.

Abstract

Universal Multimodal Retrieval (UMR) seeks any-to-any search across text and vision, yet modern embedding models remain brittle when queries require latent reasoning (e.g., resolving underspecified references or matching compositional constraints). We argue this brittleness is often data-induced: when images carry "silent" evidence and queries leave key semantics implicit, a single embedding pass must both reason and compress, encouraging spurious feature matching. We propose a data-centric framework that decouples these roles by externalizing reasoning before retrieval. Using a strong Vision--Language Model, we make implicit semantics explicit by densely captioning visual evidence in corpus entries, resolving ambiguous multimodal references in queries, and rewriting verbose instructions into concise retrieval constraints. Inference-time enhancement alone is insufficient; the retriever must be trained on these semantically dense representations to avoid distribution shift and fully exploit the added signal. Across M-BEIR, our reasoning-augmented training method yields consistent gains over strong baselines, with ablations showing that corpus enhancement chiefly benefits knowledge-intensive queries while query enhancement is critical for compositional modification requests. We publicly release our code at https://github.com/AugmentedRetrieval/ReasoningAugmentedRetrieval.

Figures (3)

• Figure 1: Overview of our reasoning-augmented multimodal representation. We use a strong VLM to (i) inject descriptive text into image-only inputs and (ii) refine captions for image--text pairs, making key visual semantics explicit. This externalizes implicit reasoning, allowing the retriever to focus on compression into robust embeddings rather than on-the-fly visual inference. Red highlights denote the matching components on both the query and corpus sides with the help of VLM-generated enhancements.
• Figure 2: In this entry of Fashion200K, the query asks for a description matching the image, which Qwen accurately captioned with keywords such as "black dress" and "flared sleeves". This allowed the retrieval of multiple correct results, indicating how the one-ground-truth design is flawed. Furthermore, the ground truth's "multicolored" description is simply untrue, reinforcing our observation over these benchmarks' low quality nature.
• Figure 3: Qualitative comparison on CIRR-7. The user's instruction requests a modification of "more focus on its head." The baseline model, relying on implicit visual features, exhibits a strong bias toward low-level visual similarity, retrieving images with matching poses (sleeping bodies) rather than the requested semantic change. In contrast, our enhanced model leverages dense corpus captions (e.g., "close-up," "profile view") to successfully align the modification instruction with the correct target.