Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models

Chuan Qin, Xin Chen, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, Boya Pan, Han Wu, Chengzan Li, Yuanchun Zhou, Hui Xiong, Hengshu Zhu

TL;DR

SciHorizon addressing the AI-for-Science readiness gap proposes a unified framework to evaluate AI-ready scientific data and LLM capabilities. It defines four data dimensions (Quality, FAIRness, Explainability, Compliance) and 16 LLM capabilities (Knowledge, Understanding, Reasoning, Multimodality, Values) across five disciplines, backed by ~1,600 datasets and 50+ LLMs. Benchmark results and public results (www.scihorizon.cn/en) reveal strengths and gaps in current data resources and language models, guiding data curation and model deployment for AI-driven scientific discovery. The framework enables systematic, cross-domain assessment and provides practical benchmarks for researchers and developers in AI4Science.

Abstract

In recent years, the rapid advancement of Artificial Intelligence (AI) technologies, particularly Large Language Models (LLMs), has revolutionized the paradigm of scientific discovery, establishing AI-for-Science (AI4Science) as a dynamic and evolving field. However, there is still a lack of an effective framework for the overall assessment of AI4Science, particularly from a holistic perspective on data quality and model capability. Therefore, in this study, we propose SciHorizon, a comprehensive assessment framework designed to benchmark the readiness of AI4Science from both scientific data and LLM perspectives. First, we introduce a generalizable framework for assessing AI-ready scientific data, encompassing four key dimensions: Quality, FAIRness, Explainability, and Compliance-which are subdivided into 15 sub-dimensions. Drawing on data resource papers published between 2018 and 2023 in peer-reviewed journals, we present recommendation lists of AI-ready datasets for Earth, Life, and Materials Sciences, making a novel and original contribution to the field. Concurrently, to assess the capabilities of LLMs across multiple scientific disciplines, we establish 16 assessment dimensions based on five core indicators Knowledge, Understanding, Reasoning, Multimodality, and Values spanning Mathematics, Physics, Chemistry, Life Sciences, and Earth and Space Sciences. Using the developed benchmark datasets, we have conducted a comprehensive evaluation of over 50 representative open-source and closed source LLMs. All the results are publicly available and can be accessed online at www.scihorizon.cn/en.

SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models

TL;DR

SciHorizon addressing the AI-for-Science readiness gap proposes a unified framework to evaluate AI-ready scientific data and LLM capabilities. It defines four data dimensions (Quality, FAIRness, Explainability, Compliance) and 16 LLM capabilities (Knowledge, Understanding, Reasoning, Multimodality, Values) across five disciplines, backed by ~1,600 datasets and 50+ LLMs. Benchmark results and public results (www.scihorizon.cn/en) reveal strengths and gaps in current data resources and language models, guiding data curation and model deployment for AI-driven scientific discovery. The framework enables systematic, cross-domain assessment and provides practical benchmarks for researchers and developers in AI4Science.

Abstract

In recent years, the rapid advancement of Artificial Intelligence (AI) technologies, particularly Large Language Models (LLMs), has revolutionized the paradigm of scientific discovery, establishing AI-for-Science (AI4Science) as a dynamic and evolving field. However, there is still a lack of an effective framework for the overall assessment of AI4Science, particularly from a holistic perspective on data quality and model capability. Therefore, in this study, we propose SciHorizon, a comprehensive assessment framework designed to benchmark the readiness of AI4Science from both scientific data and LLM perspectives. First, we introduce a generalizable framework for assessing AI-ready scientific data, encompassing four key dimensions: Quality, FAIRness, Explainability, and Compliance-which are subdivided into 15 sub-dimensions. Drawing on data resource papers published between 2018 and 2023 in peer-reviewed journals, we present recommendation lists of AI-ready datasets for Earth, Life, and Materials Sciences, making a novel and original contribution to the field. Concurrently, to assess the capabilities of LLMs across multiple scientific disciplines, we establish 16 assessment dimensions based on five core indicators Knowledge, Understanding, Reasoning, Multimodality, and Values spanning Mathematics, Physics, Chemistry, Life Sciences, and Earth and Space Sciences. Using the developed benchmark datasets, we have conducted a comprehensive evaluation of over 50 representative open-source and closed source LLMs. All the results are publicly available and can be accessed online at www.scihorizon.cn/en.

Paper Structure

This paper contains 39 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Data AI-Readiness
  4. LLM Benchmarking
  5. SciHorizon Framework
  6. Scientific Data Assessment
  7. Quality
  8. FAIRness
  9. Explainability
  10. Compliance
  11. Assessment Practices
  12. LLM Assessment
  13. Knowledge
  14. Understanding
  15. Reasoning
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Overview of the SciHorizon platform.
  • Figure 2: Radar charts of top LLMs’ performance across disciplines. This figure illustrates the top 5 LLMs’ capabilities in six evaluation dimensions: Overall, Knowledge, Understanding, Reasoning, Multimodality, and Value. Each chart displays performance across five disciplines: Mathematics, Physics, Chemistry, Life Sciences (Life), and Earth and Space Sciences (Earth).