AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research

TL;DR

The paper argues that the Science of Science (SoS) can be transformed by AI into automated large-scale pattern discovery, moving from traditional, manual analyses toward AI-driven, end-to-end SoS research. It defines AI for SoS (AI4SoS) as a meta-level, cross-disciplinary discipline that aims to automate data processing, analysis, simulation, and validation of scientific patterns, illustrated by a preliminary multi-agent system. A five-level autonomy framework is proposed to chart progress from non-automated to fully automated SoS discovery, with detailed discussion of advantages in forecasting technology trajectories and understanding research-society dynamics. The work also outlines core challenges—data imbalance, system construction, evaluation, and explainability—and provides a proof-of-concept environment that simulates large-scale scientific collaboration, demonstrating potential but highlighting substantial work needed for robust, ethical deployment and real-world impact.

Abstract

The Science of Science (SoS) explores the mechanisms underlying scientific discovery, and offers valuable insights for enhancing scientific efficiency and fostering innovation. Traditional approaches often rely on simplistic assumptions and basic statistical tools, such as linear regression and rule-based simulations, which struggle to capture the complexity and scale of modern research ecosystems. The advent of artificial intelligence (AI) presents a transformative opportunity for the next generation of SoS, enabling the automation of large-scale pattern discovery and uncovering insights previously unattainable. This paper offers a forward-looking perspective on the integration of Science of Science with AI for automated research pattern discovery and highlights key open challenges that could greatly benefit from AI. We outline the advantages of AI over traditional methods, discuss potential limitations, and propose pathways to overcome them. Additionally, we present a preliminary multi-agent system as an illustrative example to simulate research societies, showcasing AI's ability to replicate real-world research patterns and accelerate progress in Science of Science research.

Figures (7)

• Figure 1: An illustration comparing human-driven and AI-driven research processes in the SoS, highlighting step-by-step differences across four key stages in order: data processing, data analysis, system simulation, and pattern validation.
• Figure 2: An overview of the five progressively advancing levels of autonomy in AI4SoS, with more green areas indicating that higher levels correspond to greater degrees of autonomy. Current research is primarily at Level 2 or below, with very limited work at Level 3, while fully automated SoS discovery remains in the prospective stage.
• Figure 3: The overview of our preliminary multi-agent system for scientific collaboration simulation. We place the simulation within a community of scientists. After a scientist leads his/her team in submitting a paper, it undergoes peer review. If accepted, it is added to the reference database and can be cited by other scientists in subsequent epochs. Due to varying author information, the citation count of the final research output differs, then we can analyze the correlation between them—understanding the dynamics of research organizations, which is important in the field of SoS.
• Figure 4: The time taken for a complete scientific collaboration with agents of different scales. A simulation of a million-agent society takes only one week.
• Figure 5: The statistics of team sizes for papers published between 2002 and 2009 in the OAG, with the red fitting line revealing that the distribution follows an exponential pattern.