MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

TL;DR

The results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.

Abstract

Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.

Figures (42)

• Figure 1: Comparison with existing visual question answering benchmarks. Existing benchmarks often suffer from three key limitations: (i) ambiguous evidence that leads to multiple possible answers, (ii) retrieval restricted to a single modality, and (iii) small candidate pools (often limited to a single image, document, or video). In contrast, MultiHaystack provides questions grounded in uniquely verifiable evidence over a large-scale multimodal corpus of 46K+ items spanning documents, images, and videos, requiring both modality selection and fine-grained reasoning.
• Figure 2: Performance on MultiHaystack. "Gold in Top-1/5" directly provides answer-containing files; "Single-Modality" and "Cross-Modality" require retrieval within one or across multiple modalities.
• Figure 3: Examples of six tasks in MultiHaystack: Visual Parsing & Positioning (spatial layouts), Contextual Understanding (embedded text), Video Temporal Reasoning (motion/order), Statistical Reasoning (charts/tables), Metadata Identification (affiliations/timestamps), and Factual Knowledge Retrieval (corpus-grounded facts).
• Figure 4: Benchmark construction pipeline. MultiHaystack is built in four stages: collecting diverse multimodal sources, generating specific QA pairs, filtering for unique and grounded answers, and enriching with data. This design ensures coverage across six tasks (\ref{['fig:task-examples']}) and overcomes the unimodal, small-scale, or ambiguous limitations of prior benchmarks.
• Figure 5: Top-$k$ ablation analysis for MLLMs integrated with E5-V.