Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

TL;DR

This work tackles the problem of inferring correct dependencies to ensure generated repositories are executable by introducing DI-Bench, a large-scale repository-level benchmark spanning Python, C#, Rust, and JavaScript. It combines textual accuracy with CI-based execution evaluation, leveraging real GitHub Actions pipelines to assess end-to-end executability. The benchmark comprises 581 verified repos (387 regular, 194 large) and reveals that even state-of-the-art models achieve only $42.9\%$ executability, with performance deteriorating as repository size and dependency counts grow. The findings highlight hallucination and metadata quality as key bottlenecks and provide a foundation for future methods that jointly optimize dependency inference and execution viability in end-to-end AI-assisted software synthesis.

Abstract

Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

TL;DR

This work tackles the problem of inferring correct dependencies to ensure generated repositories are executable by introducing DI-Bench, a large-scale repository-level benchmark spanning Python, C#, Rust, and JavaScript. It combines textual accuracy with CI-based execution evaluation, leveraging real GitHub Actions pipelines to assess end-to-end executability. The benchmark comprises 581 verified repos (387 regular, 194 large) and reveals that even state-of-the-art models achieve only executability, with performance deteriorating as repository size and dependency counts grow. The findings highlight hallucination and metadata quality as key bottlenecks and provide a foundation for future methods that jointly optimize dependency inference and execution viability in end-to-end AI-assisted software synthesis.

Abstract

Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.
Paper Structure (36 sections, 1 equation, 14 figures, 7 tables)

This paper contains 36 sections, 1 equation, 14 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dependency Inference
  4. DI-Bench
  5. Statistics & Features
  6. Dataset Construction
  7. Repository Crawling.
  8. Test Job Locating.
  9. Execution Validating.
  10. Dependency Masking.
  11. Experiment Setup
  12. Baseline Methods.
  13. Metrics.
  14. Models.
  15. Experimental Results
  16. ...and 21 more sections

Figures (14)

  • Figure 1: An example of Python project dependencies.
  • Figure 2: An example of incorrectly identifying dependencies used in code.
  • Figure 3: CI-based curation pipeline for DI-Bench.
  • Figure 4: Distribution of failure categories (GPT-4o, All-In-One setting, Python).
  • Figure 5: Execution pass rate w.r.t dependency count.
  • ...and 9 more figures