Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents

Bhanu Prakash Vangala, Ali Adibifar, Tanu Malik, Ashish Gehani

TL;DR

This study exposes a reproducibility crisis in AI-generated code, revealing that only 68.3% of 300 projects run in a clean environment using dependencies specified by the model, while runtime dependencies explode by about 13.5×. It introduces a three-layer dependency framework (claimed, working, runtime) and the executable reliability metric to quantify reproducibility, and shows that most failures stem from code-generation errors rather than missing dependencies. The work demonstrates language- and agent-specific differences (Python most reproducible, Java hardest; Gemini excels at Python, Claude at Java) and highlights substantial hidden costs in debugging AI-generated code. The findings call for reproducibility-centric benchmarks and LLM training regimes that incorporate complete dependency closures and execution-aware testing to make AI-powered coding genuinely reliable for research and production use.

Abstract

The rise of Large Language Models (LLMs) as coding agents promises to accelerate software development, but their impact on generated code reproducibility remains largely unexplored. This paper presents an empirical study investigating whether LLM-generated code can be executed successfully in a clean environment with only OS packages and using only the dependencies that the model specifies. We evaluate three state-of-the-art LLM coding agents (Claude Code, OpenAI Codex, and Gemini) across 300 projects generated from 100 standardized prompts in Python, JavaScript, and Java. We introduce a three-layer dependency framework (distinguishing between claimed, working, and runtime dependencies) to quantify execution reproducibility. Our results show that only 68.3% of projects execute out-of-the-box, with substantial variation across languages (Python 89.2%, Java 44.0%). We also find a 13.5 times average expansion from declared to actual runtime dependencies, revealing significant hidden dependencies.

AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents

TL;DR

This study exposes a reproducibility crisis in AI-generated code, revealing that only 68.3% of 300 projects run in a clean environment using dependencies specified by the model, while runtime dependencies explode by about 13.5×. It introduces a three-layer dependency framework (claimed, working, runtime) and the executable reliability metric to quantify reproducibility, and shows that most failures stem from code-generation errors rather than missing dependencies. The work demonstrates language- and agent-specific differences (Python most reproducible, Java hardest; Gemini excels at Python, Claude at Java) and highlights substantial hidden costs in debugging AI-generated code. The findings call for reproducibility-centric benchmarks and LLM training regimes that incorporate complete dependency closures and execution-aware testing to make AI-powered coding genuinely reliable for research and production use.

Abstract

The rise of Large Language Models (LLMs) as coding agents promises to accelerate software development, but their impact on generated code reproducibility remains largely unexplored. This paper presents an empirical study investigating whether LLM-generated code can be executed successfully in a clean environment with only OS packages and using only the dependencies that the model specifies. We evaluate three state-of-the-art LLM coding agents (Claude Code, OpenAI Codex, and Gemini) across 300 projects generated from 100 standardized prompts in Python, JavaScript, and Java. We introduce a three-layer dependency framework (distinguishing between claimed, working, and runtime dependencies) to quantify execution reproducibility. Our results show that only 68.3% of projects execute out-of-the-box, with substantial variation across languages (Python 89.2%, Java 44.0%). We also find a 13.5 times average expansion from declared to actual runtime dependencies, revealing significant hidden dependencies.
Paper Structure (29 sections, 19 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 29 sections, 19 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Reproducibility Crisis in AI-Generated Code
  4. Benchmarks Missing Reproducibility
  5. Dependency Complexity and Reproducibility
  6. Gap in Reproducibility Evaluation
  7. Methodology
  8. Problem Formulation
  9. Three-Layer Dependency Analysis Framework
  10. Dataset Construction
  11. Experimental Infrastructure
  12. Environment Standardization
  13. Dependency Capture Tools
  14. Iterative Resolution Protocol
  15. Experimental Results
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Example prompt template used for all 300 code generation requests. The template explicitly emphasizes reproducibility and completeness of dependencies.
  • Figure 2: Language-specific success rates reveal ecosystem complexity impacts.
  • Figure 3: Success rate heatmap reveals agent specializations. Claude excels at Java (80%), Gemini achieves perfect Python (100%), while all agents struggle with Java except Claude.
  • Figure 4: Distribution of completeness gaps. Most projects (87%) have correct dependencies, but 13% require manual debugging to identify missing packages.
  • Figure 5: Runtime dependency explosion showing the gap between claimed (agent-declared) and runtime (actually installed) dependencies. Java shows a massive 9.5× multiplier, while JavaScript surprisingly shows almost no expansion (1.0×).
  • ...and 1 more figures