Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Executable Knowledge Graphs for Replicating AI Research

Yujie Luo, Zhuoyun Yu, Xuehai Wang, Yuqi Zhu, Ningyu Zhang, Lanning Wei, Lun Du, Da Zheng, Huajun Chen

TL;DR

The paper addresses the reproducibility bottleneck in AI research where essential implementation details are often missing. It introduces Executable Knowledge Graphs (xKG), a modular, hierarchical knowledge base that fuses textual paper knowledge with executable code to enable faithful replication. xKG is constructed automatically from arXiv papers and GitHub repositories, linking Paper, Technique, and Code nodes via structural and implementation edges, and is used to ground high-level planning and low-level implementation within multiple agent frameworks. Empirical results on PaperBench show substantial gains (up to ~10.9%) across configurations, demonstrating that structured, executable knowledge can significantly improve automated replication efforts; a codebase is released to support adoption and extension.

Abstract

Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code will released at https://github.com/zjunlp/xKG.

Executable Knowledge Graphs for Replicating AI Research

TL;DR

The paper addresses the reproducibility bottleneck in AI research where essential implementation details are often missing. It introduces Executable Knowledge Graphs (xKG), a modular, hierarchical knowledge base that fuses textual paper knowledge with executable code to enable faithful replication. xKG is constructed automatically from arXiv papers and GitHub repositories, linking Paper, Technique, and Code nodes via structural and implementation edges, and is used to ground high-level planning and low-level implementation within multiple agent frameworks. Empirical results on PaperBench show substantial gains (up to ~10.9%) across configurations, demonstrating that structured, executable knowledge can significantly improve automated replication efforts; a codebase is released to support adoption and extension.

Abstract

Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code will released at https://github.com/zjunlp/xKG.
Paper Structure (25 sections, 1 equation, 5 figures, 7 tables)

This paper contains 25 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Executable Knowledge Graphs
  3. Preliminary
  4. Design Formulation
  5. Executable Knowledge Graph Construction
  6. Corpus Curation
  7. Hierarchical Graph Construction
  8. Using Executable Knowledge Graphs
  9. Experiments
  10. Settings
  11. Main Results
  12. Further Analysis
  13. Code-based structured knowledge aids AI research replication.
  14. Successful reproduction hinges on retrieved code quality.
  15. xKG Transforms Agents from Scaffolding to Implementation.
  16. ...and 10 more sections

Figures (5)

  • Figure 1: xKG is constructed automatically from arXiv papers and GitHub repositories (Examples at Appendix \ref{['app:xkg-case']}).
  • Figure 2: Further study on Code Node quality.
  • Figure 3: Case Study on MU-DPO. A comparison of performance with and without xKG on IterativeAgent.
  • Figure 4: Average performance gain per paper.
  • Figure 5: An example of xKG data storage. Paper Nodes are stored as JSON files, with technique and code nodes embedded as structured dictionaries, where key-value pairs are used to create a one-to-one mapping representing the implementation relationship.