Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models

Jing Yu, Yuqi Tang, Kehua Feng, Mingyang Rao, Lei Liang, Zhiqiang Zhang, Mengshu Sun, Wen Zhang, Qiang Zhang, Keyan Ding, Huajun Chen

TL;DR

SciCUEval introduces a comprehensive, multimodal benchmark to evaluate large language models on scientific context understanding. It spans ten domain-specific sub-datasets across biology, chemistry, physics, biomedicine, and materials science, and integrates unstructured text, structured tables, and knowledge graphs to assess four core competencies: relevant information identification, information-absence detection, multi-source information integration, and context-aware inference. The dataset construction combines question generation, noise injection, and rigorous quality control, resulting in 11,343 questions across ten sub-datasets. An extensive evaluation of 18 LLMs reveals that explicit reasoning and larger model size improve performance, but substantial gaps remain—particularly in absence detection and cross-modal synthesis—highlighting the need for stronger reasoning, uncertainty handling, and cross-modal training for scientific-context understanding.

Abstract

Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs.

SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models

TL;DR

SciCUEval introduces a comprehensive, multimodal benchmark to evaluate large language models on scientific context understanding. It spans ten domain-specific sub-datasets across biology, chemistry, physics, biomedicine, and materials science, and integrates unstructured text, structured tables, and knowledge graphs to assess four core competencies: relevant information identification, information-absence detection, multi-source information integration, and context-aware inference. The dataset construction combines question generation, noise injection, and rigorous quality control, resulting in 11,343 questions across ten sub-datasets. An extensive evaluation of 18 LLMs reveals that explicit reasoning and larger model size improve performance, but substantial gaps remain—particularly in absence detection and cross-modal synthesis—highlighting the need for stronger reasoning, uncertainty handling, and cross-modal training for scientific-context understanding.

Abstract

Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs.

Paper Structure

This paper contains 37 sections, 4 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Backgrounds and Related Works
  3. Context Understanding Tasks
  4. Context Understanding Benchmarks
  5. Datasets
  6. Evaluation Competencies
  7. Source Data Collection
  8. Scientific Domains
  9. Data Modalities
  10. Data Generation
  11. Question Generation
  12. Noise Injection
  13. Quality Control
  14. The Final Dataset
  15. Experiments
  16. ...and 22 more sections

Figures (4)

  • Figure 1: Overview of the SciCUEval dataset. It spans five scientific domains, supports three data modalities (structured tables, knowledge graphs, and unstructured text), and includes four question types. Data are collected from high-quality scientific sources. The dataset enables evaluation across four key competencies: (1) relevant information identification, (2) information-absence detection, (3) multi-source information integration, and (4) context-aware inference.
  • Figure 2: Illustration of data generation pipeline in SciCUEval, mainly consisting of question generation, noise injection, and quality control.
  • Figure 3: Performance of LLMs across four competencies on SciCUEval.
  • Figure 4: Performance of LLMs across three modalities on SciCUEval.