Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents

TL;DR

The paper systematically surveys 181 benchmarks from 461 papers to evaluate CodeLLMs and agents across the Software Development Life Cycle (SDLC). It maps benchmarks to SDLC phases, analyzes coverage and language distribution, and identifies critical gaps, notably underrepresentation in requirements engineering and software design, with Python prevailing in benchmarks. The authors synthesize ten key findings and propose actionable directions—ranging from standardized requirements benchmarks to cross-phase, multimodal, and multi-agent evaluations—to bridge the gap between theoretical capabilities and real-world SE deployment. The work emphasizes end-to-end, realistic, and non-functional evaluation as essential for progressing CodeLLMs and agents in industrial software development, and provides a foundation for future benchmark development and benchmarking practices.

Abstract

Code large language models (CodeLLMs) and agents have shown great promise in tackling complex software engineering tasks.Compared to traditional software engineering methods, CodeLLMs and agents offer stronger abilities, and can flexibly process inputs and outputs in both natural and code. Benchmarking plays a crucial role in evaluating the capabilities of CodeLLMs and agents, guiding their development and deployment. However, despite their growing significance, there remains a lack of comprehensive reviews of benchmarks for CodeLLMs and agents. To bridge this gap, this paper provides a comprehensive review of existing benchmarks for CodeLLMs and agents, studying and analyzing 181 benchmarks from 461 relevant papers, covering the different phases of the software development life cycle (SDLC). Our findings reveal a notable imbalance in the coverage of current benchmarks, with approximately 60% focused on the software development phase in SDLC, while requirements engineering and software design phases receive minimal attention at only 5% and 3%, respectively. Additionally, Python emerges as the dominant programming language across the reviewed benchmarks. Finally, this paper highlights the challenges of current research and proposes future directions, aiming to narrow the gap between the theoretical capabilities of CodeLLMs and agents and their application in real-world scenarios.

Figures (7)

• Figure 1: Taxonomy of CodeLLM benchmarks across the SDLC. This figure categorizes existing CodeLLM benchmarks according to different phases of SDLC. The numbers indicate how many benchmarks correspond to each phase or task. Entries marked with an asterisk (*) denote benchmarks that can be used to evaluate the capabilities of CodeLLM agents. The blank spaces indicate that there are no relevant benchmarks for the current task.
• Figure 2: Research process for literature collection. This process includes keywords summary, publication search, publication filtering, and snowball expansion.
• Figure 3: The phases and their core tasks within the SDLC.
• Figure 4: Usage frequency of current main benchmarks
• Figure 5: Distribution of publication years for benchmarks across all phases of the SDLC