ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?

TL;DR

ReplicationBench evaluates AI agents on end-to-end replication of astrophysics research papers, aiming to measure faithfulness and correctness in realistic, data-driven scientific workflows. The framework decomposes papers into tasks with ground-truth outcomes, collaborates with authors for validation, and uses automated tolerance-based grading alongside qualitative expert reviews. A core dataset of 20 papers (111 tasks) and ReplicationBench-Plus (11 papers, 58 tasks) enables scalable, expert-validated assessment; frontier LLMs achieve around 22% average scores, revealing persistent challenges such as lack of persistence, conceptual errors, and execution gaps. The work demonstrates the feasibility and value of paper-scale replication benchmarks for AI in scientific research and outlines directions for improving agent reliability and extending the approach to other data-driven domains.

Abstract

Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use agents for novel research, we must first assess the underlying faithfulness and correctness of their work. To evaluate agents as research assistants, we introduce ReplicationBench, an evaluation framework that tests whether agents can replicate entire research papers drawn from the astrophysics literature. Astrophysics, where research relies heavily on archival data and computational study while requiring little real-world experimentation, is a particularly useful testbed for AI agents in scientific research. We split each paper into tasks which require agents to replicate the paper's core contributions, including the experimental setup, derivations, data analysis, and codebase. Each task is co-developed with the original paper authors and targets a key scientific result, enabling objective evaluation of both faithfulness (adherence to original methods) and correctness (technical accuracy of results). ReplicationBench is extremely challenging for current frontier language models: even the best-performing language models score under 20%. We analyze ReplicationBench trajectories in collaboration with domain experts and find a rich, diverse set of failure modes for agents in scientific research. ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks, reveals insights about agent performance generalizable to other domains of data-driven science, and provides a scalable framework for measuring AI agents' reliability in scientific research.

Figures (7)

• Figure 1: ReplicationBench evaluates AI agents on end-to-end astrophysics research paper replication. For each paper, we supply the manuscript, dataset, and execution metadata, and a complete set of expert-written tasks. We evaluate LLM agents in multi-turn code execution environments and conduct both automatic and expert human grading of traces.
• Figure 2: Three illustrative tasks of varying difficulties. Some instructions are edited for brevity.
• Figure 3: Unweighted ReplicationBench scores, with tasks binned by difficulty.
• Figure 4: To clarify the meaning of technical execution failures, we provide annotated examples of different kinds of agent errors. The following are representative snippets of agent traces featuring technical execution failures, lightly edited for brevity.
• Figure 5: Author annotations on MUSE_outflows replication attempt by Sonnet 3.7.