ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

TL;DR

ReplicatorBench presents an end-to-end benchmark for evaluating AI agents’ ability to replicate social and behavioral science claims using new data. By decomposing replication into Extraction, Generation, and Interpretation and grounding tasks in human expert SCORE replication reports, it enables fine-grained, stage-wise evaluation across 1,568 checkpoints for 19 cases. The baseline ReplicatorAgent demonstrates that current LLMs can design and execute computational experiments but struggle with acquiring new data and aligning final replication judgments with human experts. The framework highlights the need for improved data-retrieval tooling and planning capabilities to realize autonomous, real-world research replication, and it provides a public codebase to spur further development. Overall, ReplicatorBench shifts replication benchmarking from final outcomes to process-oriented assessments that better reflect real-world replication challenges in SBS domains.

Abstract

The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of this task, testing agents' ability to reproduce or replicate research outcomes when having access to the code and data. This setting, while foundational, (1) fails to capture the inconsistent availability of new data for replication as opposed to reproduction, and (2) lacks ground-truth diversity by focusing only on reproducible papers, thereby failing to evaluate an agent's ability to identify non-replicable research. Furthermore, most benchmarks only evaluate outcomes rather than the replication process. In response, we introduce ReplicatorBench, an end-to-end benchmark, including human-verified replicable and non-replicable research claims in social and behavioral sciences for evaluating AI agents in research replication across three stages: (1) extraction and retrieval of replication data; (2) design and execution of computational experiments; and (3) interpretation of results, allowing a test of AI agents' capability to mimic the activities of human replicators in real world. To set a baseline of AI agents' capability, we develop ReplicatorAgent, an agentic framework equipped with necessary tools, like web search and iterative interaction with sandboxed environments, to accomplish tasks in ReplicatorBench. We evaluate ReplicatorAgent across four underlying large language models (LLMs), as well as different design choices of programming language and levels of code access. Our findings reveal that while current LLM agents are capable of effectively designing and executing computational experiments, they struggle with retrieving resources, such as new data, necessary to replicate a claim. All code and data are publicly available at https://github.com/CenterForOpenScience/llm-benchmarking.

Figures (20)

• Figure 1: Given a research paper and a focal claim, ReplicatorBench decomposes the replication process into three stages. (1) Extraction assesses the agent's ability to gather relevant information about the claim and retrieve data resources for replication; (2) Generation assesses the computational capacity of an agent to generate and execute code for replication; (3) Interpretation assesses the agent's ability to interpret computational outputs and make conclusions regarding the focal claim's replicability.
• Figure 2: The Generation-Execution stage has two main phases: build and debug, and run and debug. We developed iterative debugging to fix issues and improve performance.
• Figure 3: Performance of GPT-4o ReplicatorAgent in a Python-only setting compared to Native setting. LLMEval scores are reported for design, execution, and interpretation stage with error bars calculated as a 95% confidence interval of the mean. Macro F1 Scores are reported for the final replication outcome (criteria met or unmet).
• Figure 4: Performance of GPT-5 ReplicatorAgent with and without access to human-written replication code. LLMEval scores are reported for design, execution, and interpretation stage with error bars calculated as a 95% confidence interval of the mean. Macro F1 Scores are reported for final replication outcome (criteria met or unmet).
• Figure 5: Execution trace for resolving a Docker build failure caused by an incompatible SciPy version.