Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Fangda Ye, Yuxin Hu, Pengxiang Zhu, Yibo Li, Ziqi Jin, Yao Xiao, Yibo Wang, Lei Wang, Zhen Zhang, Lu Wang, Yue Deng, Bin Wang, Yifan Zhang, Liangcai Su, Xinyu Wang, He Zhao, Chen Wei, Qiang Ren, Bryan Hooi, An Bo, Shuicheng Yan, Lidong Bing

Abstract

Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.

MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome

Abstract

Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.

Paper Structure

This paper contains 69 sections, 7 equations, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Query Collection and Verification
  3. Benchmark Overview
  4. Domain Coverage.
  5. Task Distribution.
  6. Cross-Distribution.
  7. User-Derived Query Curation
  8. Privacy-Preserving Processing.
  9. Classification and Routing.
  10. Difficulty-Stratified Rewriting.
  11. Automated Query Generation
  12. Trend-Grounded Generation.
  13. Three-Stage Filtering.
  14. Quality Verification
  15. Assembly.
  16. ...and 54 more sections

Figures (5)

  • Figure 1: Model performance comparison on 70 text-only deep research tasks across three dimensions.
  • Figure 2: Overview of query construction pipeline.
  • Figure 3: Overview of the query distribution. (a) Joint distribution of 12 domains and 10 task types. (b) Domain distribution. (c) Task type distribution. N = 100 (70 text-only, 30 multimodal).
  • Figure 4: Overview of the evaluation pipeline.
  • Figure 5: Relationship between synthesis quality, factuality, and statement-level precision across different systems. Left: synthesis quality vs. factuality score. Right: total number of generated statements vs. right ratio. Each point represents a system. The gray dashed lines denote linear regression fits, illustrating a weak positive correlation between synthesis quality and factuality, and a negative correlation between statement volume and precision.