Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

TL;DR

This work delivers an independent, end-to-end evaluation of Sakana's AI Scientist, revealing substantial gaps in novelty assessment, experiment execution, and manuscript quality, while documenting notable gains in automation speed and low-cost production. It shows that, despite ambitious promises, current systems require significant human oversight and struggle with methodological soundness and reliable evaluation. The study emphasizes risks such as AI-generated mass submissions and integrity concerns, and it provides a concrete roadmap—benchmarks, transparency logs, and standardized attribution—to responsibly advance ARI research. Overall, the AI Scientist demonstrates progress toward AI-assisted scientific workflows but remains a powerful but imperfect assistant rather than a fully autonomous scientific agent.

Abstract

A major step toward Artificial General Intelligence (AGI) and Super Intelligence is AI's ability to autonomously conduct research - what we term Artificial Research Intelligence (ARI). If machines could generate hypotheses, conduct experiments, and write research papers without human intervention, it would transform science. Sakana recently introduced the 'AI Scientist', claiming to conduct research autonomously, i.e. they imply to have achieved what we term Artificial Research Intelligence (ARI). The AI Scientist gained much attention, but a thorough independent evaluation has yet to be conducted. Our evaluation of the AI Scientist reveals critical shortcomings. The system's literature reviews produced poor novelty assessments, often misclassifying established concepts (e.g., micro-batching for stochastic gradient descent) as novel. It also struggles with experiment execution: 42% of experiments failed due to coding errors, while others produced flawed or misleading results. Code modifications were minimal, averaging 8% more characters per iteration, suggesting limited adaptability. Generated manuscripts were poorly substantiated, with a median of five citations, most outdated (only five of 34 from 2020 or later). Structural errors were frequent, including missing figures, repeated sections, and placeholder text like 'Conclusions Here'. Some papers contained hallucinated numerical results. Despite these flaws, the AI Scientist represents a leap forward in research automation. It generates full research manuscripts with minimal human input, challenging expectations of AI-driven science. Many reviewers might struggle to distinguish its work from human researchers. While its quality resembles a rushed undergraduate paper, its speed and cost efficiency are unprecedented, producing a full paper for USD 6 to 15 with 3.5 hours of human involvement, far outpacing traditional researchers.