Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis

Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Hongshuai Wang, Yongge Li, Mujie Lin, Shuwen Yang, Jiankun Wang, Mingjun Xu, Jin Huang, Xi Fang, Jiaxi Zhuang, Yuqi Yin, Yaqi Li, Changhong Chen, Zheng Cheng, Zifeng Zhao, Linfeng Zhang, Guolin Ke

TL;DR

SciAssess introduces a domain-specific benchmark for scientific literature analysis across biology, chemistry, materials, and medicine, structuring evaluation around three ability levels (memorization, comprehension, analysis & reasoning) and five modalities to test multimodal extraction and reasoning. It comprises 6,938 questions across 27 tasks with rigorous quality control, and evaluates 11 LLMs, revealing a consistent strength of OpenAI and Gemini models in higher-level tasks while exposing multimodal and PDF-structure parsing challenges. The work situates SciAssess among existing science benchmarks by highlighting the need for tasks that require interpretation of non-textual content and real-world document parsing, and it outlines a path toward broader domains and richer multimodal evaluation. Overall, SciAssess provides a comprehensive, realism-grounded framework to steer future LLM development for scientific literature analysis and knowledge extraction.

Abstract

Recent breakthroughs in Large Language Models (LLMs) have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis \& Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at \url{https://github.com/sci-assess/SciAssess}.

SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis

TL;DR

SciAssess introduces a domain-specific benchmark for scientific literature analysis across biology, chemistry, materials, and medicine, structuring evaluation around three ability levels (memorization, comprehension, analysis & reasoning) and five modalities to test multimodal extraction and reasoning. It comprises 6,938 questions across 27 tasks with rigorous quality control, and evaluates 11 LLMs, revealing a consistent strength of OpenAI and Gemini models in higher-level tasks while exposing multimodal and PDF-structure parsing challenges. The work situates SciAssess among existing science benchmarks by highlighting the need for tasks that require interpretation of non-textual content and real-world document parsing, and it outlines a path toward broader domains and richer multimodal evaluation. Overall, SciAssess provides a comprehensive, realism-grounded framework to steer future LLM development for scientific literature analysis and knowledge extraction.

Abstract

Recent breakthroughs in Large Language Models (LLMs) have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis \& Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at \url{https://github.com/sci-assess/SciAssess}.
Paper Structure (52 sections, 5 figures, 8 tables)

This paper contains 52 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Benchmark Dataset
  3. Ability Assessment Framework
  4. Scope & Task
  5. Biology
  6. Chemistry
  7. Materials
  8. Medicine
  9. Data Quality, Privacy, and Copyright
  10. Experiment
  11. Experiment Setup
  12. Results and Analysis
  13. Performances of Different Ability Levels
  14. Performance on Multimodal Contents
  15. Related Work
  16. ...and 37 more sections

Figures (5)

  • Figure 1: Overview of SciAssess. It spans over 4 sub-domains and encompasses 27 tasks.
  • Figure 2: Performance overview of leading open and closed source LLMs on SciAssess. Each column represents a scientific domain. LLMs are evaluated on multiple tasks within each domain, with task details provided in Table \ref{['tab:benchmark-stat']}. For closed source LLMs (first row), GPT-4o and GPT-4 are the leading models. For open source LLMs (second row), Llama3 and Qwen2 emerge as the top models.
  • Figure 3: Distribution of token length for questions and answers in each task.
  • Figure 4: Example of Tag to Molecule task.
  • Figure 5: Question types.