Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models

TL;DR

This work introduces VDR-Bench, a 2,000-instance, vision-centric benchmark for evaluating multimodal deep-research systems that combine visual understanding, web search, and multi-hop reasoning. It identifies two core flaws in existing benchmarks: reliance on text-only shortcuts and overly idealized retrieval conditions. A vision-first data curation pipeline and a multi-round cropped-search workflow are proposed to enforce visual evidence and realistic search dynamics. Experiments demonstrate that iterative visual search and the Multi-turn Visual Forcing paradigm substantially improve performance, providing practical guidance for building more robust Vision-DeepResearch agents.

Abstract

Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs. Second, overly idealized evaluation scenario: On the image-search side, the required information can often be obtained via near-exact matching against the full image, while the text-search side is overly direct and insufficiently challenging. To address these issues, we construct the Vision-DeepResearch benchmark (VDR-Bench) comprising 2,000 VQA instances. All questions are created via a careful, multi-stage curation pipeline and rigorous expert review, designed to assess the behavior of Vision-DeepResearch systems under realistic real-world conditions. Moreover, to address the insufficient visual retrieval capabilities of current MLLMs, we propose a simple multi-round cropped-search workflow. This strategy is shown to effectively improve model performance in realistic visual retrieval scenarios. Overall, our results provide practical guidance for the design of future multimodal deep-research systems. The code will be released in https://github.com/Osilly/Vision-DeepResearch.

Figures (4)

• Figure 1: Motivation. Existing Vision-DeepResearch benchmarks often fail to measure realistic multimodal search: many questions can be solved via text-only cues or model priors without genuine visual verification, and whole-image search frequently retrieves near-duplicate images with identifying metadata ("perfect retrieval"). VDR-Bench is designed to be visual-search–centric and to reflect real-world settings that require iterative, entity-level localization (e.g., multi-round cropping), cross-modal evidence collection, and multi-hop reasoning.
• Figure 2: Dataset composition and example instances from VDR-Bench. The figure presents the distribution across ten visual domains, along with representative question–answer examples for each category.
• Figure 3: VDR-Bench is constructed via a multi-stage, vision-centric workflow: (1) annotators manually crop salient regions and perform web-scale visual search; (2) candidate entities are extracted from retrieved results and verified through an MLLM-assisted and human checking process; (3) verified visual entities are used to generate seed VQA pairs; (4) question difficulty is expanded via knowledge-graph–based multi-hop reasoning; and (5) automatic solvability checks and human quality filtering ensure that each instance requires visual evidence, remains unambiguous, and avoids trivial or near-duplicate retrieval.
• Figure 4: Relationship between overall answer accuracy and entity-level recall on VDR-Bench. Points correspond to different models evaluated under two retrieval strategies (CIS+TS and CIS+TS+MVF), with colors indicating model families and marker shapes indicating the search mode. The plot shows a strong positive association between correctly answering questions and successfully retrieving task-relevant entities, and further indicates that MVF tends to improve both metrics across models.