Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence

Yiheng Wang, Yixin Chen, Shuo Li, Yifan Zhou, Bo Liu, Hengjian Gao, Jiakang Yuan, Jia Bu, Wanghan Xu, Yuhao Zhou, Xiangyu Zhao, Zhiwang Zhou, Fengxiang Wang, Haodong Duan, Songyang Zhang, Jun Yao, Han Deng, Yizhou Wang, Jiabei Xiao, Jiaqi Liu, Encheng Su, Yujie Liu, Weida Wang, Junchi Yao, Shenghe Zheng, Haoran Sun, Runmin Ma, Xiangchao Yan, Bo Zhang, Dongzhan Zhou, Shufei Zhang, Peng Ye, Xiaosong Wang, Shixiang Tang, Wenlong Zhang, Lei Bai

TL;DR

SciEvalKit addresses the gap in evaluating scientific intelligence by delivering an open-source, multimodal, execution-aware evaluation toolkit organized around a seven-dimension capability taxonomy and six disciplinary domains. It combines expert-aligned benchmarks with a four-layer evaluation framework and a unified prompt-prediction interface, enabling reproducible, capability-oriented assessment of LLMs and MLLMs. Key findings reveal strong knowledge-understanding capabilities but persistent weaknesses in symbolic reasoning and code generation, even among leading models, underscoring the need for tighter integration of visual grounding with domain-specific semantics. The work provides a standardized infrastructure and leaderboard to drive progress toward robust scientific problem solving in next-generation AI systems.

Abstract

We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.

SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence

TL;DR

SciEvalKit addresses the gap in evaluating scientific intelligence by delivering an open-source, multimodal, execution-aware evaluation toolkit organized around a seven-dimension capability taxonomy and six disciplinary domains. It combines expert-aligned benchmarks with a four-layer evaluation framework and a unified prompt-prediction interface, enabling reproducible, capability-oriented assessment of LLMs and MLLMs. Key findings reveal strong knowledge-understanding capabilities but persistent weaknesses in symbolic reasoning and code generation, even among leading models, underscoring the need for tighter integration of visual grounding with domain-specific semantics. The work provides a standardized infrastructure and leaderboard to drive progress toward robust scientific problem solving in next-generation AI systems.

Abstract

We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.
Paper Structure (37 sections, 17 equations, 6 figures)

This paper contains 37 sections, 17 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Benchmark Suite
  3. Core Competencies Taxonomy of Scientific Intelligence
  4. Scientific Discipline Coverage
  5. Expert-Aligned Benchmark Construction
  6. Principles of Expert‑Aligned Benchmark Design
  7. Benchmark Overview
  8. Evaluation Framework
  9. Abstraction Layer
  10. Unified interface for prompt construction and prediction
  11. Capability-Oriented Evaluation
  12. Evaluation Modes
  13. Evaluation Results
  14. Capability-Wise Leaderboard
  15. Scientific Text Capabilities
  16. ...and 22 more sections

Figures (6)

  • Figure 1: Overview of the SciEvalKit scientific intelligence evaluation framework.
  • Figure 2: Comparison of model performance on scientific versus general tasks.
  • Figure 4: Evaluation pipeline used in SciEvalKit.
  • Figure 5: Large-language-model (LLM) scientific capabilities (left) versus multimodal-language-model (MLLM) scientific capabilities (right) comparing the evaluated models on the SciEval leaderboard. Each axis reports the score (0 – 100) for one capability or scientific field; concentric rings mark 20 intervals up to the outer 100 score.
  • Figure 7: Model scores on four text-only capacities: Scientific Knowledge Understanding (Knowl. Und.), Scientific Code Generation (Code Gen.), Scientific Symbolic Reasoning (Symbolic Reason.), Scientific Hypothesis Generation (Hypoth. Gen.), and their mean (Text Overall).
  • ...and 1 more figures