Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents

Zhengbo Zhang, Jinbo Su, Zhaowen Zhou, Changtao Miao, Yuhan Hong, Qimeng Wu, Yumeng Liu, Feier Wu, Yihe Tian, Yuhao Liang, Zitong Shan, Wanke Xia, Yi-Fan Zhang, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled browsing agents to acquire and reason over multimodal information in the real world. But existing benchmarks suffer from two limitations: insufficient evaluation of visual reasoning ability and the neglect of native visual information of web pages in the reasoning chains. To address these challenges, we introduce a new benchmark for visual-native search, VisBrowse-Bench. It contains 169 VQA instances covering multiple domains and evaluates the models' visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning. These data were constructed by human experts using a multi-stage pipeline and underwent rigorous manual verification. We additionally propose an agent workflow that can effectively drive the browsing agent to actively collect and reason over visual information during the search process. We comprehensively evaluated both open-source and closed-source models in this workflow. Experimental results show that even the best-performing model, Claude-4.6-Opus only achieves an accuracy of 47.6%, while the proprietary Deep Research model, o3-deep-research only achieves an accuracy of 41.1%. The code and data can be accessed at: https://github.com/ZhengboZhang/VisBrowse-Bench

VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled browsing agents to acquire and reason over multimodal information in the real world. But existing benchmarks suffer from two limitations: insufficient evaluation of visual reasoning ability and the neglect of native visual information of web pages in the reasoning chains. To address these challenges, we introduce a new benchmark for visual-native search, VisBrowse-Bench. It contains 169 VQA instances covering multiple domains and evaluates the models' visual reasoning capabilities during the search process through multimodal evidence cross-validation via text-image retrieval and joint reasoning. These data were constructed by human experts using a multi-stage pipeline and underwent rigorous manual verification. We additionally propose an agent workflow that can effectively drive the browsing agent to actively collect and reason over visual information during the search process. We comprehensively evaluated both open-source and closed-source models in this workflow. Experimental results show that even the best-performing model, Claude-4.6-Opus only achieves an accuracy of 47.6%, while the proprietary Deep Research model, o3-deep-research only achieves an accuracy of 41.1%. The code and data can be accessed at: https://github.com/ZhengboZhang/VisBrowse-Bench
Paper Structure (31 sections, 7 figures, 3 tables)

This paper contains 31 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Browsing Agents
  4. Multimodal Browsing Benchmarks
  5. VisBrowse-Bench
  6. Data Design Principles
  7. Multimodal Information Integration:
  8. Visual Competency Enforcement:
  9. Data Collection
  10. Quality Validation
  11. Dataset Statistics
  12. Agentic Framework
  13. Discussion
  14. Experiments
  15. Experimental Setups
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Existing benchmarks have two limitations in evaluating multimodal browsing agents: 1. the semantic information of visual queries can be easily obtained through image search tools; 2. real-world browsing environments contain a wealth of multimodal information, which most benchmarks overlook. VisBrowse-Bench is designed to fuse multimodal information during the search process and ensure that visual capabilities are essential for completing the task.
  • Figure 2: (a) Overall performance of MLLMs on VisBrowse-Bench. (b) Performance of four MLLMs on seven categories.
  • Figure 3: Overview of the VisBrowse-Bench. This figure shows the example questions and answers across seven categories.
  • Figure 4: (a) Distribution of categories in VisBrowse-Bench. (b) Summary of statistics.
  • Figure 5: The percentage of times each of the five tools was used in four MLLMs: Kimi-K2.5, Gemini-3.0-Pro, Claude-4.6-Opus and GPT-5.2
  • ...and 2 more figures