HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery

TL;DR

HiSciBench addresses the fragmented landscape of scientific intelligence benchmarks by introducing a hierarchical, multidisciplinary benchmark that spans five cognitive levels from literacy to discovery and supports multimodal and cross-lingual inputs. The framework is evaluated on 8,735 instances across six disciplines using 18 diverse models, revealing strong performance on basic literacy tasks but substantial gaps in literature synthesis, cross-modal reasoning, and data-driven discovery. Key findings show that while models like GPT-5 excel at foundational tasks, they struggle with high-level synthesis and verifiable scientific grounding, highlighting epistemic and multimodal integration challenges. The benchmark provides a standardized, diagnostic tool to guide the development of more reliable and generalizable scientific AI capable of moving beyond knowledge recall toward genuine reading-to-discovery workflows.

Abstract

The rapid advancement of large language models (LLMs) and multimodal foundation models has sparked growing interest in their potential for scientific research. However, scientific intelligence encompasses a broad spectrum of abilities ranging from understanding fundamental knowledge to conducting creative discovery, and existing benchmarks remain fragmented. Most focus on narrow tasks and fail to reflect the hierarchical and multi-disciplinary nature of real scientific inquiry. We introduce \textbf{HiSciBench}, a hierarchical benchmark designed to evaluate foundation models across five levels that mirror the complete scientific workflow: \textit{Scientific Literacy} (L1), \textit{Literature Parsing} (L2), \textit{Literature-based Question Answering} (L3), \textit{Literature Review Generation} (L4), and \textit{Scientific Discovery} (L5). HiSciBench contains 8,735 carefully curated instances spanning six major scientific disciplines, including mathematics, physics, chemistry, biology, geography, and astronomy, and supports multimodal inputs including text, equations, figures, and tables, as well as cross-lingual evaluation. Unlike prior benchmarks that assess isolated abilities, HiSciBench provides an integrated, dependency-aware framework that enables detailed diagnosis of model capabilities across different stages of scientific reasoning. Comprehensive evaluations of leading models, including GPT-5, DeepSeek-R1, and several multimodal systems, reveal substantial performance gaps: while models achieve up to 69\% accuracy on basic literacy tasks, performance declines sharply to 25\% on discovery-level challenges. HiSciBench establishes a new standard for evaluating scientific Intelligence and offers actionable insights for developing models that are not only more capable but also more reliable. The benchmark will be publicly released to facilitate future research.

Figures (5)

• Figure 1: Overview of HiSciBench, a hierarchical benchmark for evaluating scientific intelligence in large language models. It covers six scientific disciplines includingmathematics, physics, chemistry, biology, geography, and astronomy—shown on the left, and five progressively structured tasks (L1–L5) on the right, encompassing Scientific Literacy, Literature Parsing, Literature QA, Review Generation, and Scientific Discovery. The hierarchy mirrors the full scientific workflow, progressing from L1 factual understanding and L2 literature parsing, to L3 contextual reasoning, L4 integrative synthesis, and finally L5 creative discovery.
• Figure 2: HiSciBench: Comprehensive Distribution Analysis. (a) Distribution by task level across cognitive hierarchies (L1--L5), with perception and reasoning tasks (L3.1) dominating the benchmark. (b) Distribution by scientific discipline, showing balanced coverage across six domains, where Biology (26.6%) and Physics (26.4%) are the largest. (c) Distribution by modality, indicating that most tasks (84.7%) involve structured image–text inputs, followed by text-only and data–text settings.
• Figure 3: Radar chart of model performance across HiSciBench tasks (L1, L2.2, L3.2, L4, and L5). GPT-5 demonstrates the most balanced performance across tasks, particularly excelling in reasoning (L3.2) and factual QA (L1). Deepseek-r1 performs competitively in cross-lingual QA but trails in multimodal and discovery-oriented tasks. S1 models are specialized for scientific research: S1-Literature is used for L4.1 (Literature Review Generation), while S1-Base-Pro (32B) is applied to all other tasks. The red dashed line indicates an ideal model achieving at least 60 points across all tasks.
• Figure 4: Radar comparison of GPT-5’s performance on L2.2: Cross-lingual Scientific Translation across four disciplines under text-only and vision-language inputs. The text-input curve achieves higher BLEU scores (37–49), while the vision-language curve consistently trails by 12–17 BLEU, revealing that visual context currently introduces semantic noise instead of aiding linguistic reasoning.
• Figure 5: Comparison of content quality and citation verifiability rate across general-purpose and specialized models on L4: Scientific Literature Review Generation. While all models achieve near-perfect content quality (88.8–99.8%), their citation verifiability remains drastically lower (17–22%), revealing an 80% factuality gap. This demonstrates that LLMs can generate coherent and well-structured scientific reviews, yet often fail to provide verifiable or authentic citations. The results suggest that surface-level fluency does not equate to factual grounding, emphasizing the need for stronger citation retrieval and source alignment mechanisms in future scientific LLMs.