MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

TL;DR

MM-BRIGHT introduces the first multimodal benchmark for reasoning-intensive retrieval, pairing 2,803 real-world StackExchange queries across 29 domains with four retrieval tasks that span text, images, and multimodal outputs. Through extensive evaluation of 18 models, the study reveals that current methods struggle to combine visual understanding with technical reasoning, often performing worse when images are present or when requiring multimodal alignment. Key findings include large headroom relative to traditional benchmarks, a predominance of essential visual content in queries, and the limited utility of image captions or reformulation for these tasks. The benchmark, its dataset, and code are released to spur development of next-generation retrieval models that better integrate visual reasoning with complex information needs.

Abstract

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.

Figures (17)

• Figure 1: To comprehensively evaluate multimodal retrieval capabilities, we systematically define four retrieval tasks of increasing multimodal complexity. These range from a text-only baseline (i) to complex multimodal-to-multimodal retrieval (iv), requiring different levels of visual reasoning and context integration.
• Figure 2: Overview of the MM-BRIGHT annotation process for Stack Exchange data. Queries are multimodal Stack Exchange posts containing text and images. Positive documents (text and/or images) are discovered by annotators using Gemini AI assistance or from links in accepted answers, then manually verified for relevance. Negative documents are mined using GPT-4o-generated search queries and entities designed to find similar but actually irrelevant content. Documents can include Wikipedia pages, blogs, articles, research papers, and technical documentation.
• Figure 3: Distribution of image types in MM-BRIGHT.
• Figure 4: Image essentiality distribution in MM-BRIGHT.
• Figure 5: Impact of image captioning on retrieval performance across different retriever types. We augment text-only queries with image captions generated by various vision-language models and measure nDCG@10 on Task 1.