Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

Wanghan Xu, Yuhao Zhou, Yifan Zhou, Qinglong Cao, Shuo Li, Jia Bu, Bo Liu, Yixin Chen, Xuming He, Xiangyu Zhao, Xiang Zhuang, Fengxiang Wang, Zhiwang Zhou, Qiantai Feng, Wenxuan Huang, Jiaqi Wei, Hao Wu, Yuejin Yang, Guangshuai Wang, Sheng Xu, Ziyan Huang, Xinyao Liu, Jiyao Liu, Cheng Tang, Wei Li, Ying Chen, Junzhi Ning, Pengfei Jiang, Chenglong Ma, Ye Du, Changkai Ji, Huihui Xu, Ming Hu, Jiangbin Zheng, Xin Chen, Yucheng Wu, Feifei Jiang, Xi Chen, Xiangru Tang, Yuchen Fu, Yingzhou Lu, Yuanyuan Zhang, Lihao Sun, Chengbo Li, Jinzhe Ma, Wanhao Liu, Yating Liu, Kuo-Cheng Wu, Shengdu Chai, Yizhou Wang, Ouwen Zhangjin, Chen Tang, Shufei Zhang, Wenbo Cao, Junjie Ren, Taoyong Cui, Zhouheng Yao, Juntao Deng, Yijie Sun, Feng Liu, Wangxu Wei, Jingyi Xu, Zhangrui Li, Junchao Gong, Zijie Guo, Zhiyu Yao, Zaoyu Chen, Tianhao Peng, Fangchen Yu, Bo Zhang, Dongzhan Zhou, Shixiang Tang, Jiaheng Liu, Fenghua Ling, Yan Lu, Yuchen Ren, Ben Fei, Zhen Zhao, Xinyu Gu, Rui Su, Xiao-Ming Wu, Weikang Si, Yang Liu, Hao Chen, Xiangchao Yan, Xue Yang, Junchi Yan, Jiamin Wu, Qihao Zheng, Chenhui Li, Zhiqiang Gao, Hao Kong, Junjun He, Mao Su, Tianfan Fu, Peng Ye, Chunfeng Song, Nanqing Dong, Yuqiang Li, Huazhu Fu, Siqi Sun, Lijing Cheng, Jintai Lin, Wanli Ouyang, Bowen Zhou, Wenlong Zhang, Lei Bai

TL;DR

This work defines Scientific General Intelligence (SGI) as an AI able to autonomously navigate the full, iterative cycle of scientific inquiry with human-like versatility. It operationalizes SGI through SGI-Bench, a four-task, workflow-aligned benchmark (Scientific Deep Research, Idea Generation, Dry/Wet Experiment, Experimental Reasoning) evaluated via an Agentic SGI framework (SGIEvalAgent) that combines scientist-aligned data construction, diverse metrics, and tool-enabled reasoning. Across open- and closed-weight models, results reveal that current LLMs/agents achieve pockets of success within individual quadrants but struggle to integrate long-horizon, numerically faithful scientific reasoning across multiple domains. The paper additionally introduces Test-Time Reinforcement Learning (TTRL) with novelty rewards to promote idea generation without ground-truth supervision and demonstrates that retrieval-based feedback can drive measurable improvements in hypothesis creativity. Together, these contributions provide a principled, scalable foundation for developing AI systems capable of genuine scientific discovery and iterative self-improvement, while highlighting key bottlenecks (retrieval latency, numerical robustness, and executable planning) and clear avenues for future research.

Abstract

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

TL;DR

This work defines Scientific General Intelligence (SGI) as an AI able to autonomously navigate the full, iterative cycle of scientific inquiry with human-like versatility. It operationalizes SGI through SGI-Bench, a four-task, workflow-aligned benchmark (Scientific Deep Research, Idea Generation, Dry/Wet Experiment, Experimental Reasoning) evaluated via an Agentic SGI framework (SGIEvalAgent) that combines scientist-aligned data construction, diverse metrics, and tool-enabled reasoning. Across open- and closed-weight models, results reveal that current LLMs/agents achieve pockets of success within individual quadrants but struggle to integrate long-horizon, numerically faithful scientific reasoning across multiple domains. The paper additionally introduces Test-Time Reinforcement Learning (TTRL) with novelty rewards to promote idea generation without ground-truth supervision and demonstrates that retrieval-based feedback can drive measurable improvements in hypothesis creativity. Together, these contributions provide a principled, scalable foundation for developing AI systems capable of genuine scientific discovery and iterative self-improvement, while highlighting key bottlenecks (retrieval latency, numerical robustness, and executable planning) and clear avenues for future research.

Abstract

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.

Paper Structure

This paper contains 92 sections, 39 equations, 34 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Scientific General Intelligence: Concept and Operational Definition
  3. Task Definition in Scientific Workflow
  4. Scientific Deep Research
  5. Idea Generation
  6. Dry/Wet Experiment
  7. Dry Experiment
  8. Wet Experiment
  9. Experimental Reasoning
  10. Multi-Dimensional Metrics
  11. Metrics of Scientific Deep Research
  12. Metrics of Idea Generation
  13. Metrics of Dry/Wet Experiment
  14. Dry Experiment
  15. Wet Experiment
  16. ...and 77 more sections

Figures (34)

  • Figure 1: Scientific General Intelligence (SGI) We define SGI as an AI that can autonomously navigate the complete, iterative cycle of scientific inquiry with the versatility and proficiency of a human scientist. The teaser illustrates the Practical Inquiry Model's four quadrants—Deliberation (synthesis and critical evaluation of knowledge), Conception (idea generation), Action (experimental execution), and Perception (interpretation)—and how SGI-Bench operationalizes them through four task categories and an agent-based evaluation paradigm, together providing a principle-grounded, measurable framework for assessing scientific intelligence.
  • Figure 2: SGI-Bench Workflow Pipeline: The end-to-end four-stage framework (Deliberation, Conception, Action, Perception) that operationalizes scientific discovery, mapping tasks to capabilities and aligning evaluation with scientist practice.
  • Figure 3: Scientific Deep Research Task: Inputs, outputs, and formulation for literature-driven quantitative inquiry combining multi-step reasoning and meta-analysis.
  • Figure 4: Idea Generation Task: Inputs, outputs, and formulation for methodology design, integrating evaluation metrics and structured implementation planning.
  • Figure 5: Dry Experiment Task: Inputs, outputs, and formulation for code-completion based computational studies with masked functions.
  • ...and 29 more figures