Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering

TL;DR

LLMs for code are reshaping software engineering, but existing evaluation benchmarks fail to capture real-world contexts, robustness, and efficiency due to data leakage and narrow metrics. The authors propose BEHELM, a holistic benchmarking infrastructure that combines a software-scenario taxonomy with multi-metric evaluation to assess LLMs across tasks, languages, and quality dimensions. They base the framework on a synthesis of workshop discussions and a survey of current benchmarks, outlining concrete recommendations for community-driven data pipelines, provenance, and contamination controls. BEHELM aims to reduce benchmark construction overhead while enabling fair, realistic, and future-proof assessment of AI-assisted software development.

Abstract

Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering.

Figures (1)

• Figure 1: Current Evaluation System vs. Infrastructure for Holistic Evaluation of LLMc .