AI Scientists Fail Without Strong Implementation Capability
Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, Yue Zhang
TL;DR
The paper addresses the gap between idea generation and actionable, verifiable research in AI Scientists, arguing that the key bottleneck is the ability to implement and verify complex experiments. It supports this claim with trend analyses, cross-domain benchmarks, and a simulated peer-review study, all pointing to a persistent verification/implementation deficit. By formalizing AI Scientists, presenting quantitative evidence, and outlining ethical and governance considerations, it highlights the practical barriers to full autonomy and guides future research toward robust execution and evaluation frameworks. The work emphasizes that bridging the implementation gap is essential for realizing truly autonomous scientific progress and calls for community-wide efforts, standardized evaluation, and responsible governance.
Abstract
The emergence of Artificial Intelligence (AI) Scientist represents a paradigm shift in scientific discovery, with large language models (LLMs) taking the lead as the primary executor in the entire scientific workflow from idea generation to experiment implementation. Recent AI Scientist studies demonstrate sufficient capabilities for independent scientific discovery, with the generated research reports gaining acceptance at the ICLR 2025 workshop and ACL 2025, arguing that a human-level AI Scientist, capable of uncovering phenomena previously unknown to humans, may be imminent. Despite this substantial progress, AI Scientist has yet to produce a groundbreaking achievement in the domain of computer science on par with automated scientific tools. Based on extensive quantitative evidence from existing benchmarks in complex engineering tasks and a systematic evaluation assess 28 research papers generated by five advanced AI Scientist systems, we argue that \textbf{the fundamental bottleneck for AI Scientists lies in their capability to execute the requisite verification procedures.} Current AI Scientist systems lack the execution capabilities needed to execute rigorous experiments and produce high-quality scientific papers. To better illustrate the root cause of this \textbf{implementation gap}, we provide an in-depth discussion on the fundamental limitations of AI Scientist. This position paper aims to call for the participants in the community to bridge the implementation gap.