Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers

Zijie Lin, Yiqing Shen, Qilin Cai, He Sun, Jinrui Zhou, Mingjun Xiao

TL;DR

This work defines the Paper-to-Code (P2C) task, which aims to convert multimodal content from ML research papers into fully executable code repositories. It introduces AutoP2C, a four-stage, LLM-based multi-agent framework that derives repository blueprints, parses text, diagrams, and tables, decomposes tasks hierarchically, and employs iterative feedback-driven implementation to produce runnable code accompanied by explanatory diagrams. The authors propose novel evaluation metrics for structural completeness and validate AutoP2C on Paper2Repo and PaperBench Code-Dev benchmarks, showing superior functional correctness and repository fidelity compared with baselines, and even surpassing original implementations in some cases. The results demonstrate notable gains in reproducibility and practical utility, with future work planned toward broader language support and enhanced user feedback for continuous improvement.

Abstract

Machine Learning (ML) research is spread through academic papers featuring rich multimodal content, including text, diagrams, and tabular results. However, translating these multimodal elements into executable code remains a challenging and time-consuming process that requires substantial ML expertise. We introduce ``Paper-to-Code'' (P2C), a novel task that transforms the multimodal content of scientific publications into fully executable code repositories, which extends beyond the existing formulation of code generation that merely converts textual descriptions into isolated code snippets. To automate the P2C process, we propose AutoP2C, a multi-agent framework based on large language models that processes both textual and visual content from research papers to generate complete code repositories. Specifically, AutoP2C contains four stages: (1) repository blueprint extraction from established codebases, (2) multimodal content parsing that integrates information from text, equations, and figures, (3) hierarchical task decomposition for structured code generation, and (4) iterative feedback-driven debugging to ensure functionality and performance. Evaluation on a benchmark of eight research papers demonstrates the effectiveness of AutoP2C, which can successfully generate executable code repositories for all eight papers, while OpenAI-o1 or DeepSeek-R1 can only produce runnable code for one paper. The code is available at https://github.com/shoushouyu/Automated-Paper-to-Code.

AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers

TL;DR

This work defines the Paper-to-Code (P2C) task, which aims to convert multimodal content from ML research papers into fully executable code repositories. It introduces AutoP2C, a four-stage, LLM-based multi-agent framework that derives repository blueprints, parses text, diagrams, and tables, decomposes tasks hierarchically, and employs iterative feedback-driven implementation to produce runnable code accompanied by explanatory diagrams. The authors propose novel evaluation metrics for structural completeness and validate AutoP2C on Paper2Repo and PaperBench Code-Dev benchmarks, showing superior functional correctness and repository fidelity compared with baselines, and even surpassing original implementations in some cases. The results demonstrate notable gains in reproducibility and practical utility, with future work planned toward broader language support and enhanced user feedback for continuous improvement.

Abstract

Machine Learning (ML) research is spread through academic papers featuring rich multimodal content, including text, diagrams, and tabular results. However, translating these multimodal elements into executable code remains a challenging and time-consuming process that requires substantial ML expertise. We introduce ``Paper-to-Code'' (P2C), a novel task that transforms the multimodal content of scientific publications into fully executable code repositories, which extends beyond the existing formulation of code generation that merely converts textual descriptions into isolated code snippets. To automate the P2C process, we propose AutoP2C, a multi-agent framework based on large language models that processes both textual and visual content from research papers to generate complete code repositories. Specifically, AutoP2C contains four stages: (1) repository blueprint extraction from established codebases, (2) multimodal content parsing that integrates information from text, equations, and figures, (3) hierarchical task decomposition for structured code generation, and (4) iterative feedback-driven debugging to ensure functionality and performance. Evaluation on a benchmark of eight research papers demonstrates the effectiveness of AutoP2C, which can successfully generate executable code repositories for all eight papers, while OpenAI-o1 or DeepSeek-R1 can only produce runnable code for one paper. The code is available at https://github.com/shoushouyu/Automated-Paper-to-Code.
Paper Structure (21 sections, 5 equations, 3 figures, 4 tables)

This paper contains 21 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Large Language Models for Code Generation
  4. Multi-Agent for Code Generation
  5. Methods
  6. Problem Formulation
  7. Method Overview
  8. Repository Blueprint Extraction
  9. Multimodal Content Parsing
  10. Hierarchical Task Decomposition
  11. Iterative Feedback-Driven Implementation
  12. Experiments
  13. Evaluation Metrics
  14. Benchmark and Baseline
  15. Experiment Settings
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Illustration of the "Paper-to-Code" (P2C) task. It involves processing multimodal content, such as text, images, and tables, from an academic paper to automatically generate a complete, executable code repository together with explanatory diagrams.
  • Figure 2: Overview of the AutoP2C framework. It contains four stage, namely (1) repository blueprint extraction, which analyzes established ML repositories to generate a standardized blueprint for future code organization; (2) multimodal content parsing, which processes PDF papers into a unified, implementation-focused representation; (3) hierarchical task decomposition, which generates structured implementation plans; and (4) iterative feedback-driven implementation, which transforms task descriptions into an executable code repository.
  • Figure 3: Case study illustrates the complete workflow of AutoP2C for the EDoRA: ... nasiri2025edora paper. The process begins with (1) extracting multimodal content (text, images, tables) from the original research paper using MinerU wang2024mineru, followed by (2) multimodal content parsing to produce refined, structured content with cross-modal relationships preserved. Based on this processed multimodal information, AutoP2C then performs (3) hierarchical task decomposition to generate detailed file descriptions with explicit references to paper sections. These descriptions guide (4) the implementation of functional code that accurately captures the paper's algorithmic specifications, culminating in (5) the generation of explanatory diagrams that visualize component relationships in the resulting code repository.