Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents

Peizhou Huang, Zixuan Zhong, Zhongwei Wan, Donghao Zhou, Samiul Alam, Xin Wang, Zexin Li, Zhihao Dou, Li Zhu, Jing Xiong, Chaofan Tao, Yan Xu, Dimitrios Dimitriadis, Tuo Zhang, Mi Zhang

TL;DR

MMDR-Bench addresses the gap in end-to-end multimodal deep research evaluation by introducing 140 image-text tasks across 21 domains and a tri-partite evaluation pipeline (FLAE, TRACE, MOSAIC) that jointly measures report quality, citation-grounded fidelity, and text-visual integrity. The framework enforces strict visual-grounding fidelity via Visual Evidence Fidelity and provides task-adaptive, interpretable diagnostics beyond a single score. Experiments with 25 state-of-the-art models reveal persistent trade-offs between prose quality, citation discipline, and multimodal grounding, identifying multimodal integrity as a primary bottleneck. The work offers reproducible benchmarks, detailed ablations, and insights into how tool use and retrieval interact with backbone capabilities to influence end-to-end deep-research performance.

Abstract

Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.

MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents

TL;DR

MMDR-Bench addresses the gap in end-to-end multimodal deep research evaluation by introducing 140 image-text tasks across 21 domains and a tri-partite evaluation pipeline (FLAE, TRACE, MOSAIC) that jointly measures report quality, citation-grounded fidelity, and text-visual integrity. The framework enforces strict visual-grounding fidelity via Visual Evidence Fidelity and provides task-adaptive, interpretable diagnostics beyond a single score. Experiments with 25 state-of-the-art models reveal persistent trade-offs between prose quality, citation discipline, and multimodal grounding, identifying multimodal integrity as a primary bottleneck. The work offers reproducible benchmarks, detailed ablations, and insights into how tool use and retrieval interact with backbone capabilities to influence end-to-end deep-research performance.

Abstract

Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.
Paper Structure (34 sections, 8 equations, 13 figures, 9 tables)

This paper contains 34 sections, 8 equations, 13 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Collection
  4. Evaluation Methodology
  5. FLAE: Formula-LLM Adaptive Evaluation
  6. TRACE: Trustworthy Retrieval-Aligned Citation Evaluation
  7. MOSAIC: Multimodal Support-Aligned Integrity Check
  8. Experiments
  9. Experimental Setup
  10. Main Results and Findings
  11. Fine-Grained Domains Analysis
  12. Human Consistency Check
  13. Robustness of Judge Model and Weights
  14. Conclusion
  15. Appendix
  16. ...and 19 more sections

Figures (13)

  • Figure 1: Overall MMDR-Bench score (0--100; higher is better) on 140 tasks for representative tool-using LMMs and Deep Research systems, ranked by score.
  • Figure 2: MMDR-Bench evaluates multimodal deep research abilities at both integrated and atomic levels.
  • Figure 3: Task distribution of MMDR-Bench.
  • Figure 4: Two example tasks from MMDR-Bench.
  • Figure 5: The MMDR-Bench evaluation pipeline. Reports are processed through parallel FLAE and TRACE modules, followed by a gated MOSAIC stage.
  • ...and 8 more figures