Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li

TL;DR

DevEval addresses the discrepancy between existing code-generation benchmarks and practical software development by introducing a manually annotated benchmark aligned with real-world repositories. It collects 1,874 samples from 117 repositories across 10 domains, each sample comprising six components and annotated by 13 developers with natural-language requirements and precise dependency references, enabling a repository-level coding task evaluated via test cases. The evaluation framework uses $Pass@k$ for functional correctness and $Recall@k$ for dependency recall to compare eight LLMs under three input settings, revealing substantial gaps between standard benchmarks and real-world coding performance. The work advances toward more realistic evaluation and prompts future directions, including multilingual expansion and context-retrieval improvements.

Abstract

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

TL;DR

DevEval addresses the discrepancy between existing code-generation benchmarks and practical software development by introducing a manually annotated benchmark aligned with real-world repositories. It collects 1,874 samples from 117 repositories across 10 domains, each sample comprising six components and annotated by 13 developers with natural-language requirements and precise dependency references, enabling a repository-level coding task evaluated via test cases. The evaluation framework uses for functional correctness and for dependency recall to compare eight LLMs under three input settings, revealing substantial gaps between standard benchmarks and real-world coding performance. The work advances toward more realistic evaluation and prompts future directions, including multilingual expansion and context-retrieval improvements.

Abstract

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.
Paper Structure (19 sections, 2 equations, 6 figures, 5 tables)

This paper contains 19 sections, 2 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Benchmark - DevEval
  3. Overview
  4. Task Definition
  5. Evaluation Metrics
  6. Features of DevEval
  7. Benchmark Construction
  8. Experiments
  9. Studied LLMs
  10. Experimental Settings
  11. Evaluation
  12. Main Results
  13. Empirical Lessons
  14. Discussion
  15. Related Work
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Examples of standalone and non-standalone functions. Dependencies are highlighted, i.e., yellow: intra-class dependencies, green: intra-file dependencies, and blue: cross-file dependencies.
  • Figure 2: An overview of DevEval. Each sample consists of six components.
  • Figure 3: The process of collecting DevEval.
  • Figure 4: A failed case of gpt-3.5 in the Local File (Completion) setting.
  • Figure 5: Pass@1 of gpt-4 on different program types.
  • ...and 1 more figures