MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval

TL;DR

MR$^2$-Bench addresses the gap in evaluating retrieval systems on reasoning-intensive multimodal queries that require deep visual-text reasoning. It introduces three meta-tasks and 12 sub-tasks spanning knowledge retrieval, visual illustration, and visual relation reasoning, with 1,309 queries across six domains and support for interleaved multi-image documents. Across 11 embedding models, plus caption augmentation and various reranking strategies, the benchmark reveals that current models struggle on MR$^2$-Bench, with significant performance gaps compared to existing benchmarks, and that reasoning-based approaches and multimodal rerankers offer meaningful gains. The results underscore the importance of visual reasoning and multimodal integration for realistic retrieval, and the authors provide data and code publicly to accelerate progress. For example, despite high Recall@1 scores on some benchmarks, Seed1.6-Embedding achieves only $9.91$ on MR$^2$-Bench, highlighting the increased challenge. The work thus sets a new, more demanding standard for multimodal retrieval research and points to productive directions including query rewriting and advanced reranking.

Abstract

Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios. Existing benchmarks primarily probe surface-level semantic correspondence (e.g., object-text matching) while failing to assess the deeper reasoning required to capture complex relationships between visual and textual information. To address this gap, we introduce MR$^2$-Bench, a reasoning-intensive benchmark for multimodal retrieval. MR$^2$-Bench presents the following critical values: 1) all tasks are reasoning-driven, going beyond shallow matching to effectively assess models' capacity for logical, spatial, and causal inference; 2) it features diverse multimodal data, such as natural images, diagrams, and visual puzzles, enabling comprehensive evaluation across content types; 3) it supports complex queries and documents containing multiple images and covers diverse retrieval scenarios, more accurately reflecting real-world applications. Our benchmark contains 1,309 curated queries, derived either from manual collection and annotation or from selective consolidation of public datasets. Despite achieving strong results on existing benchmarks, current state-of-the-art models still struggle on MR$^2$-Bench: for example, the leading Seed1.6-Embedding model attains a Recall@1 of 77.78 on MMEB, but only 9.91 on MR$^2$-Bench. This substantial performance gap highlights both the increased challenge posed by our benchmark and the pressing need for further advances in reasoning-intensive multimodal retrieval. The dataset and evaluation code will be made publicly available at https://github.com/VectorSpaceLab/MR2-Bench.

Figures (4)

• Figure 1: Visualized Examples of MR$^2$-Bench: Sub-task illustrations from three meta-tasks, with 3 out of 6 shown for the multimodal knowledge retrieval task.
• Figure 2: Reranking performance on MR$^{2}$-Bench with Seed-1.6-Embedding as the base retriever.
• Figure 3: Prompt used by GPT-5 for query rewriting.
• Figure 4: Prompt used by MLLMs to score query-candidate pairs after reasoning.