Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

TL;DR

The paper presents the first comprehensive systematic literature review of 291 SE benchmarks used to evaluate large language models, structured around three questions: what benchmarks exist, how they are constructed, and what challenges remain. It documents diverse SE tasks (requirements/design, coding, testing, AIOps, maintenance, quality management) and analyzes benchmark construction methods, evaluation metrics, and central limitations such as data contamination and language bias. The authors highlight key findings, including the prominence of coding-assistant benchmarks and the rise of real-world, competition-like SWE-bench variants, while calling for more diverse languages, multi-metric evaluation, and dynamic benchmarks aligned with industrial practice. They also release an open-source repo (LLM4SEBenchmarks) to curate benchmarks and foster fair comparison, signaling a practical impact on future SE evaluation of LLMs. Overall, the work provides a structured, actionable overview of benchmark maturity, gaps, and opportunities to guide robust, real-world evaluation of SE-facing LLMs.

Abstract

Large language models (LLMs) are gaining increasing popularity in software engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available, how benchmarks are constructed, and the future outlook for these benchmarks. We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOPs, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

Figures (4)

• Figure 1: Overview of this survey.
• Figure 2: Number of benchmarks over the years.
• Figure 3: Benchmark Distribution by Task.
• Figure 4: Count of Benchmark by Programming Language and Task (Programming languages with fewer than three benchmarks are excluded).