Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

Dhruv Trehan, Paras Chopra

TL;DR

This study investigates how far contemporary LLM-driven agents can autonomously generate ML research artifacts with minimal scaffolding. Using a six-module autonomous pipeline, the authors executed four seed ideas across three ML subdomains, with only one idea (AS-1) successfully completed and published at Agents4Science 2025 after both AI and human review. They identify six recurring failure modes—bias from training data, implementation drift, memory/context decay, overexcitement, insufficient domain intelligence, and weak scientific taste—and propose four design principles to improve robustness, including verification, failure planning, memory management, and comprehensive logging. The work underscores the current limits of autonomous scientific discovery, emphasizes the need for human-in-the-loop oversight and open-science artifacts, and outlines concrete paths for scaling, benchmarking, and systematic evaluation in future research.

Abstract

We report a case study of four end-to-end attempts to autonomously generate ML research papers using a pipeline of six LLM agents mapped to stages of the scientific workflow. Of these four, three attempts failed during implementation or evaluation. One completed the pipeline and was accepted to Agents4Science 2025, an experimental inaugural venue that required AI systems as first authors, passing both human and multi-AI review. From these attempts, we document six recurring failure modes: bias toward training data defaults, implementation drift under execution pressure, memory and context degradation across long-horizon tasks, overexcitement that declares success despite obvious failures, insufficient domain intelligence, and weak scientific taste in experimental design. We conclude by discussing four design principles for more robust AI-scientist systems, implications for autonomous scientific discovery, and we release all prompts, artifacts, and outputs at https://github.com/Lossfunk/ai-scientist-artefacts-v1

Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

TL;DR

This study investigates how far contemporary LLM-driven agents can autonomously generate ML research artifacts with minimal scaffolding. Using a six-module autonomous pipeline, the authors executed four seed ideas across three ML subdomains, with only one idea (AS-1) successfully completed and published at Agents4Science 2025 after both AI and human review. They identify six recurring failure modes—bias from training data, implementation drift, memory/context decay, overexcitement, insufficient domain intelligence, and weak scientific taste—and propose four design principles to improve robustness, including verification, failure planning, memory management, and comprehensive logging. The work underscores the current limits of autonomous scientific discovery, emphasizes the need for human-in-the-loop oversight and open-science artifacts, and outlines concrete paths for scaling, benchmarking, and systematic evaluation in future research.

Abstract

We report a case study of four end-to-end attempts to autonomously generate ML research papers using a pipeline of six LLM agents mapped to stages of the scientific workflow. Of these four, three attempts failed during implementation or evaluation. One completed the pipeline and was accepted to Agents4Science 2025, an experimental inaugural venue that required AI systems as first authors, passing both human and multi-AI review. From these attempts, we document six recurring failure modes: bias toward training data defaults, implementation drift under execution pressure, memory and context degradation across long-horizon tasks, overexcitement that declares success despite obvious failures, insufficient domain intelligence, and weak scientific taste in experimental design. We conclude by discussing four design principles for more robust AI-scientist systems, implications for autonomous scientific discovery, and we release all prompts, artifacts, and outputs at https://github.com/Lossfunk/ai-scientist-artefacts-v1
Paper Structure (17 sections, 7 figures, 8 tables)

This paper contains 17 sections, 7 figures, 8 tables.

Table of Contents

  1. Problem Definition and System Overview
  2. Our Attempts and Outcomes
  3. Observed Failure Modes and Mitigation
  4. Bias on Training Data
  5. Implementation Drift
  6. Memory and Context Issues
  7. Overexcitement and Eureka Instinct
  8. Lack of Sufficient Domain Intelligence
  9. Lack of Scientific Taste
  10. Design Takeaways for AI Scientist Systems
  11. Discussion and Broader Implications
  12. Limitations and Future Work
  13. Deep Dives: Research Ideas and Anecdotes
  14. MARL-1: Zero-shot Coordination in Multi-Agent RL
  15. WM-1: Differentiable Planning in Stochastic World Models
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Autonomous Research Pipeline: High-level diagram showing the interaction between the six agent modules and the shared file system artifacts (idea.md to paper_outline.md) used to maintain context.
  • Figure 2: Agent Prompt Template: How repository state, tools, and process guidelines are shared in the system prompt of each agent module.
  • Figure 3: The Selection Funnel: From 135+ papers to 4 candidates. Only one (AS-1) survived execution constraints.
  • Figure 4: Training Data Bias Pattern: Systematic override of prompt instructions by memorized patterns across long contexts.
  • Figure 5: Implementation Drift Pattern: Simplification of proposed implementations during execution barriers in Idea WM-1 .
  • ...and 2 more figures