DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search

TL;DR

DeepMMSearch-R1 introduces a multimodal LLM capable of on-demand, multi-turn web searches with dynamic query generation for both image and text tools, augmented by an intermediate image-cropping step. A two-stage training regime—cold-start supervised fine-tuning on the DeepMMSearchVQA dataset followed by online reinforcement learning via GRPO—teaches the model when and how to search, and how to reason over retrieved content. The DeepMMSearchVQA dataset provides multi-turn, tool-annotated VQA samples that mix real web-retrieved content with structured reasoning to train self-reflection and self-correction. Across several knowledge-intensive benchmarks, DeepMMSearch-R1 achieves state-of-the-art or competitive performance against strong baselines, demonstrating the value of integrating crop-based image search and iterative query refinement in multimodal web-search-enabled LLMs. The work offers insights into tool usage, search efficiency, and maintaining general visual reasoning while interacting with dynamic web information, with potential applications in intelligent web agents and digital assistants.

Abstract

Multimodal Large Language Models (MLLMs) in real-world applications require access to external knowledge sources and must remain responsive to the dynamic and ever-changing real-world information in order to address information-seeking and knowledge-intensive user queries. Existing approaches, such as retrieval augmented generation (RAG) methods, search agents, and search equipped MLLMs, often suffer from rigid pipelines, excessive search calls, and poorly constructed search queries, which result in inefficiencies and suboptimal outcomes. To address these limitations, we present DeepMMSearch-R1, the first multimodal LLM capable of performing on-demand, multi-turn web searches and dynamically crafting queries for both image and text search tools. Specifically, DeepMMSearch-R1 can initiate web searches based on relevant crops of the input image making the image search more effective, and can iteratively adapt text search queries based on retrieved information, thereby enabling self-reflection and self-correction. Our approach relies on a two-stage training pipeline: a cold start supervised finetuning phase followed by an online reinforcement learning optimization. For training, we introduce DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated pipeline intermixed with real-world information from web search tools. This dataset contains diverse, multi-hop queries that integrate textual and visual information, teaching the model when to search, what to search for, which search tool to use and how to reason over the retrieved information. We conduct extensive experiments across a range of knowledge-intensive benchmarks to demonstrate the superiority of our approach. Finally, we analyze the results and provide insights that are valuable for advancing multimodal web-search.

Figures (8)

• Figure 1: Unlike previous baselines, which lack self-reflection, self-correction, and cropped image-based search, the proposed DeepMMSearch-R1 is capable of performing on-demand, multi-turn web searches with enhanced image search that incorporates an intermediate cropping tool to select the most relevant region of an image. It demonstrates self-reflection and self-correction abilities, iteratively refining its text queries to better navigate noisy real-world web information. The model outperforms other baselines, notably GPT-4o, and is competitive with the GPT-o3 model.
• Figure 2: (top)DeepMMSearchVQA Data Generation Pipeline: It begins by passing a question--image pair $(q, i)$ to Gemini, which produces reasoning and concludes with an action tag. We then apply checks A, B, and C: if either A or B fails, the example is discarded; if C passes, the final answer is reached and the example is saved. Otherwise, the pipeline invokes a search tool guided by the action tag. This tool retrieves the top-$k$ web results, which are then summarized and fed back into Gemini, incorporating web-retrieved information in its context for subsequent turns in the reasoning process. (bottom)DeepMMSearchVQA Statistics: Knowledge taxonomy, Distribution of examples across different numbers of conversational turns, Proportion of questions with text search, image search and cropped image search.
• Figure 3: (left) Impact of self-reflection, self-correction and cropped image search on performance. (right) Effect of the ratio of search-required to search-free data, and of sampling strategies when curating SFT data.
• Figure 4: Tool usage statistics after supervised finetuning and online RL on DynVQA and OK-VQA benchmarks.
• Figure 5: After SFT, the model performs unnecessary cropping. RL training refines the tool-use behavior, making it more efficient and invoking tools only when required.