Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents

Zijie Lin, Qilin Cai, Liang Shen, Mingjun Xiao

TL;DR

Reproducibility of scientific results is impeded by the lack of systematic verification during automated paper-to-code reproduction. The authors introduce prompt-free collaborative agents—a verification agent and a refinement agent—that rely solely on original system prompts to validate and improve outputs, integrated into the Paper2Code workflow. Across PaperBench Code-Dev and Paper2CodeBench, the approach yields substantial improvements in reproduction accuracy and completeness, and demonstrates robustness against Self-Refine while reducing iteration costs relative to RePro. This work offers a practical, scalable direction for automated code generation in research pipelines.

Abstract

Automated paper reproduction has emerged as a promising approach to accelerate scientific research, employing multi-step workflow frameworks to systematically convert academic papers into executable code. However, existing frameworks often lack mechanisms to verify and refine the outputs at each generation step, or rely heavily on manually designed prompts for self-refinement, which limits their adaptability and scalability. To address these limitations, we propose a prompt-free collaborative agent framework that automatically enhances the quality of paper-to-code generation. Our approach employs two collaborative agents: a verification agent that examines whether the outputs at each step satisfy the requirements specified in the corresponding system prompt, and a refinement agent that revises the outputs based on the identified issues. Unlike previous methods that require human experts to craft specific refinement prompts for each step, our framework achieves automatic verification and improvement by leveraging only the original system prompts. We integrate our collaborative agents into the Paper2Code framework and conduct comprehensive experiments on PaperBench Code-Dev and Paper2CodeBench datasets. Experimental results demonstrate that our approach significantly improves the accuracy and completeness of reproduced code, achieving performance gains of approximately 15\% and 13\%, respectively, compared to the baseline without our agents. Furthermore, comparative experiments against Self-Refine validate the robustness and consistency of our prompt-free approach across different datasets.

Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents

TL;DR

Reproducibility of scientific results is impeded by the lack of systematic verification during automated paper-to-code reproduction. The authors introduce prompt-free collaborative agents—a verification agent and a refinement agent—that rely solely on original system prompts to validate and improve outputs, integrated into the Paper2Code workflow. Across PaperBench Code-Dev and Paper2CodeBench, the approach yields substantial improvements in reproduction accuracy and completeness, and demonstrates robustness against Self-Refine while reducing iteration costs relative to RePro. This work offers a practical, scalable direction for automated code generation in research pipelines.

Abstract

Automated paper reproduction has emerged as a promising approach to accelerate scientific research, employing multi-step workflow frameworks to systematically convert academic papers into executable code. However, existing frameworks often lack mechanisms to verify and refine the outputs at each generation step, or rely heavily on manually designed prompts for self-refinement, which limits their adaptability and scalability. To address these limitations, we propose a prompt-free collaborative agent framework that automatically enhances the quality of paper-to-code generation. Our approach employs two collaborative agents: a verification agent that examines whether the outputs at each step satisfy the requirements specified in the corresponding system prompt, and a refinement agent that revises the outputs based on the identified issues. Unlike previous methods that require human experts to craft specific refinement prompts for each step, our framework achieves automatic verification and improvement by leveraging only the original system prompts. We integrate our collaborative agents into the Paper2Code framework and conduct comprehensive experiments on PaperBench Code-Dev and Paper2CodeBench datasets. Experimental results demonstrate that our approach significantly improves the accuracy and completeness of reproduced code, achieving performance gains of approximately 15\% and 13\%, respectively, compared to the baseline without our agents. Furthermore, comparative experiments against Self-Refine validate the robustness and consistency of our prompt-free approach across different datasets.

Paper Structure

This paper contains 11 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Verification Agent: Prompt-Free Output Validation
  4. Refinement Agent: Automatic Output Improvement
  5. Application to Paper2Code Workflow
  6. Experiment
  7. Datasets
  8. Baselines
  9. Experiment Settings
  10. Main Results
  11. Conclusion

Figures (2)

  • Figure 1: Comparison of the planning stage in PaperCoder between the self-refine and auto-refine methods. The left panel illustrates the self-refine approach, where each planning step requires human-crafted prompts for verification and refinement. The right panel presents our proposed auto-refine method, which employs an automated optimization loop. In this approach, a unified agent automatically extracts prompts to verify and refine the plan at each stage, eliminating the need for manual intervention.
  • Figure 2: Per-paper performance comparison on PaperBench between our planning optimization approach (orange) and Self-Refine (blue). Relative scores are calculated against the Paper2Code baseline.