Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches

TL;DR

Computational reproducibility in social science remains challenging due to missing dependencies and fragile execution contexts. The authors compare two AI-enabled repair strategies—prompt-based large language models and autonomous agent-based repair—using a synthetic benchmark of five reproducible R studies with injected failures in Dockerized environments. Prompt-based repairs achieve 31-79% success depending on context and error complexity, while agent-based workflows reach 69-96% success, demonstrating the superior ability of agents to diagnose, modify, and re-run analyses. The study provides a controlled, post-publication repair framework for direct method comparison and highlights practical implications for integrating AI-assisted reproducibility tools into research workflows. Overall, agent-based approaches with execution access can substantially reduce manual debugging and improve reproducibility across diverse error types in computational social science.

Abstract

Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data. In practice, missing packages, fragile file paths, version conflicts, or incomplete logic frequently cause analyses to fail, even when materials are shared. This study investigates whether large language models and AI agents can automate the diagnosis and repair of such failures, making computational results easier to reproduce and verify. We evaluate this using a controlled reproducibility testbed built from five fully reproducible R-based social science studies. Realistic failures were injected, ranging from simple issues to complex missing logic, and two automated repair workflows were tested in clean Docker environments. The first workflow is prompt-based, repeatedly querying language models with structured prompts of varying context, while the second uses agent-based systems that inspect files, modify code, and rerun analyses autonomously. Across prompt-based runs, reproduction success ranged from 31-79 percent, with performance strongly influenced by prompt context and error complexity. Complex cases benefited most from additional context. Agent-based workflows performed substantially better, with success rates of 69-96 percent across all complexity levels. These results suggest that automated workflows, especially agent-based systems, can significantly reduce manual effort and improve reproduction success across diverse error types. Unlike prior benchmarks, our testbed isolates post-publication repair under controlled failure modes, allowing direct comparison of prompt-based and agent-based approaches.

Figures (5)

• Figure 1: Overview of the experimental framework. First, we construct a synthetic benchmark dataset by injecting systematic failures into reproducible R studies to create a controlled testbed. Then, we evaluate two repair strategies: an iterative, prompt-based workflow and an autonomous agent-based workflow, both operating in isolated Docker environments. Finally, results were aggregated by error category and workflow to enable comparative statistical analyses.
• Figure 2: Overview of the synthetic reproducibility benchmark.(a) Classification of error types plotted against error diversity and repair difficulty, ranging from simple execution errors (Category A) to complex structural logic gaps (Category C). (b) The structural organization of a single synthetic test case, containing the original publication as context, input data, support scripts, and the analysis script with multiple injected errors.
• Figure 3: Prompt-based LLM Repair Workflow. Success rates (%) of GPT-4o, Gemini 2.5 Pro, and Qwen3-Coder across error categories (y-axis) and prompt types (x-axis). Each prompt level shows a grouped bar chart of model performance for that error–prompt combination, illustrating how error complexity and contextual information affect automated reproducibility.
• Figure 4: Qwen3-Coder Performance in Agent-based and Prompt-based Workflow. Percentage of successfully reproduced cases using Qwen3-Coder across error categories. Bars show agent-based workflow performance with OpenCode and Claude Code, while dashed horizontal lines indicate for comparison the success rates of the prompt-based workflow using Qwen3-Coder with the full-context prompt.
• Figure 5: Improvement of Agent-based Workflows over Prompt-based. Absolute improvement in reproducibility success rates achieved by agent-based workflows relative to the prompt-based results using Qwen3-Coder. Bars show the difference in percentage points, meaning the absolute increase in success rates for each agent (OpenCode and Claude Code) compared to the prompt-based workflow.