Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

Yuhao Zhou, Yiheng Wang, Xuming He, Ao Shen, Ruoyao Xiao, Zhiwei Li, Qiantai Feng, Zijie Guo, Yuejin Yang, Hao Wu, Wenxuan Huang, Jiaqi Wei, Dan Si, Xiuqi Yao, Jia Bu, Haiwen Huang, Manning Wang, Tianfan Fu, Shixiang Tang, Ben Fei, Dongzhan Zhou, Fenghua Ling, Yan Lu, Siqi Sun, Chenhui Li, Guanjie Zheng, Jiancheng Lv, Wenlong Zhang, Lei Bai

TL;DR

The Scientists' First Exam (SFE) introduces a novel, multilingual benchmark to rigorously assess scientific cognitive abilities of multimodal LLMs across perception, understanding, and reasoning. By spanning 66 tasks over five disciplines and using expert-authored VQA pairs, SFE reveals nuanced model capabilities and gaps, notably showing stronger performance on higher-level reasoning (L3) for newer models and greater difficulty in astronomy. The evaluation of 16 MLLMs across open and closed weight families demonstrates substantial room for improvement and highlights the importance of data-scale balance with model size. Overall, SFE provides a granular, domain-specific framework to drive progress in AI-assisted scientific discovery and benchmarking across diverse scientific frontiers.

Abstract

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning

TL;DR

The Scientists' First Exam (SFE) introduces a novel, multilingual benchmark to rigorously assess scientific cognitive abilities of multimodal LLMs across perception, understanding, and reasoning. By spanning 66 tasks over five disciplines and using expert-authored VQA pairs, SFE reveals nuanced model capabilities and gaps, notably showing stronger performance on higher-level reasoning (L3) for newer models and greater difficulty in astronomy. The evaluation of 16 MLLMs across open and closed weight families demonstrates substantial room for improvement and highlights the importance of data-scale balance with model size. Overall, SFE provides a granular, domain-specific framework to drive progress in AI-assisted scientific discovery and benchmarking across diverse scientific frontiers.

Abstract

Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries.

Paper Structure

This paper contains 89 sections, 5 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SFE Dataset and Tasks
  4. Data Collection
  5. Tasks
  6. Experiments and Evaluations
  7. Main Results
  8. Analysis
  9. Conclusion
  10. PrismaX Team
  11. Tasks Overview
  12. Tasks Description
  13. Question Format
  14. Responses Evaluation
  15. Subtask Description
  16. ...and 74 more sections

Figures (5)

  • Figure 1: Overview of the Scientists' First Exam (SFE) benchmark.
  • Figure 2: Data collection framework of SFE. First, we define 18 scientific directions based on the scientific frontier and domain experts. Building upon this structure, we invite experts to propose tasks and contribute raw data based on three cognitive levels. Finally, we employ visualization techniques and further engage experts to annotate the resulting benchmark.
  • Figure 3: Pass@k scores of four state-of-the-art MLLMs on SFE. Closed-weight MLLMs demonstrate superior initial performance and greater scalability than open weight MLLMs.
  • Figure 4: The scaling law of model size on SFE. Clearly, the amount of scientific data has not been scaled proportionally with the increase in model size, resulting in minor overfit for QwenVL-2.5-72B.
  • Figure 5: Case studies on SFE across different cognitive levels.