Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M$^3$Searcher: Modular Multimodal Information Seeking Agency with Retrieval-Oriented Reasoning

Xiaohan Yu, Chao Feng, Lang Mei, Chong Chen

TL;DR

M^3Searcher tackles the multimodal information-seeking gap by decoupling retrieval from answer derivation in MRAG systems. It introduces a modular, retrieval-oriented RL framework trained on MMSearchVQA, a dataset designed to encourage deep, multi-hop multimodal search with explicit evidence. The approach combines a dedicated information-seeking agent with a modality-agnostic answer generator, optimized via GRPO and multi-objective rewards that emphasize formatting, answer quality, and faithful retrieval. Empirical results show strong cross-domain transfer, robust tool coordination, and substantial gains over prompt-engineered and end-to-end baselines, highlighting the practical potential of retrieval-driven multimodal agents for real-world information synthesis.

Abstract

Recent advances in DeepResearch-style agents have demonstrated strong capabilities in autonomous information acquisition and synthesize from real-world web environments. However, existing approaches remain fundamentally limited to text modality. Extending autonomous information-seeking agents to multimodal settings introduces critical challenges: the specialization-generalization trade-off that emerges when training models for multimodal tool-use at scale, and the severe scarcity of training data capturing complex, multi-step multimodal search trajectories. To address these challenges, we propose M$^3$Searcher, a modular multimodal information-seeking agent that explicitly decouples information acquisition from answer derivation. M$^3$Searcher is optimized with a retrieval-oriented multi-objective reward that jointly encourages factual accuracy, reasoning soundness, and retrieval fidelity. In addition, we develop MMSearchVQA, a multimodal multi-hop dataset to support retrieval centric RL training. Experimental results demonstrate that M$^3$Searcher outperforms existing approaches, exhibits strong transfer adaptability and effective reasoning in complex multimodal tasks.

M$^3$Searcher: Modular Multimodal Information Seeking Agency with Retrieval-Oriented Reasoning

TL;DR

M^3Searcher tackles the multimodal information-seeking gap by decoupling retrieval from answer derivation in MRAG systems. It introduces a modular, retrieval-oriented RL framework trained on MMSearchVQA, a dataset designed to encourage deep, multi-hop multimodal search with explicit evidence. The approach combines a dedicated information-seeking agent with a modality-agnostic answer generator, optimized via GRPO and multi-objective rewards that emphasize formatting, answer quality, and faithful retrieval. Empirical results show strong cross-domain transfer, robust tool coordination, and substantial gains over prompt-engineered and end-to-end baselines, highlighting the practical potential of retrieval-driven multimodal agents for real-world information synthesis.

Abstract

Recent advances in DeepResearch-style agents have demonstrated strong capabilities in autonomous information acquisition and synthesize from real-world web environments. However, existing approaches remain fundamentally limited to text modality. Extending autonomous information-seeking agents to multimodal settings introduces critical challenges: the specialization-generalization trade-off that emerges when training models for multimodal tool-use at scale, and the severe scarcity of training data capturing complex, multi-step multimodal search trajectories. To address these challenges, we propose MSearcher, a modular multimodal information-seeking agent that explicitly decouples information acquisition from answer derivation. MSearcher is optimized with a retrieval-oriented multi-objective reward that jointly encourages factual accuracy, reasoning soundness, and retrieval fidelity. In addition, we develop MMSearchVQA, a multimodal multi-hop dataset to support retrieval centric RL training. Experimental results demonstrate that MSearcher outperforms existing approaches, exhibits strong transfer adaptability and effective reasoning in complex multimodal tasks.
Paper Structure (32 sections, 7 equations, 8 figures, 1 table)

This paper contains 32 sections, 7 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. M$^3$Searcher
  3. Task Formulation
  4. Decoupled Agentic MRAG
  5. Multimodal Tools Implementation
  6. Decoupled Multi-turn Rollout
  7. Multi-Objective Rewrad Modeling
  8. Format Reward
  9. Answer Reward
  10. Information Retrieval Reward
  11. RL Training
  12. MMSearchVQA Dataset
  13. Experiments
  14. Experimental Setup
  15. Datasets
  16. ...and 17 more sections

Figures (8)

  • Figure 1: The architecture of M$^3$Searcher.
  • Figure 2: The MMSearchVQA data construction pipeline.
  • Figure 3: Overview of MMSearchVQA dataset statistics. The left figure summarizes the domain distribution, and the right figure reports the distribution of question difficulty levels and reasoning hops.
  • Figure 4: Ablation study.
  • Figure 5: Training dynamics of reward and rollout turn counts with and without the information-retrieval reward.
  • ...and 3 more figures