Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

TL;DR

<3-5 sentence high-level summary> This paper investigates whether LLMs can replace human evaluators for SE tasks by evaluating seven LLM-as-a-judge methods plus two evaluation-tuned LLMs across code translation, code generation, and code summarization. It finds that output-based methods with large LLMs often achieve the strongest alignment with human judgments, approaching near-human performance on translation and generation but struggling on summarization, while embedding- and probability-based methods generally underperform and lack generalizability. Pairwise comparisons are notably unreliable, even for the best methods, underscoring limitations for relying on LLMs to perform direct comparisons. The study provides a first empirical SE-focused meta-evaluation dataset, reveals task-dependent patterns in method performance, and offers practical guidance and future directions for SE evaluation research and tooling.

Abstract

Recently, large language models (LLMs) have been deployed to tackle various software engineering (SE) tasks like code generation, significantly advancing the automation of SE tasks. However, assessing the quality of these LLM-generated code and text remains challenging. The commonly used Pass@k metric necessitates extensive unit tests and configured environments, demands a high labor cost, and is not suitable for evaluating LLM-generated text. Conventional metrics like BLEU, which measure only lexical rather than semantic similarity, have also come under scrutiny. In response, a new trend has emerged to employ LLMs for automated evaluation, known as LLM-as-a-judge. These LLM-as-a-judge methods are claimed to better mimic human assessment than conventional metrics without relying on high-quality reference answers. Nevertheless, their exact human alignment in SE tasks remains unexplored. In this paper, we empirically explore LLM-as-a-judge methods for evaluating SE tasks, focusing on their alignment with human judgments. We select seven LLM-as-a-judge methods that utilize general-purpose LLMs, alongside two LLMs specifically fine-tuned for evaluation. After generating and manually scoring LLM responses on three recent SE datasets of code translation, code generation, and code summarization, we then prompt these methods to evaluate each response. Finally, we compare the scores generated by these methods with human evaluation. The results indicate that output-based methods reach the highest Pearson correlation of 81.32 and 68.51 with human scores in code translation and generation, achieving near-human evaluation, noticeably outperforming ChrF++, one of the best conventional metrics, at 34.23 and 64.92. Such output-based methods prompt LLMs to output judgments directly, and exhibit more balanced score distributions that resemble human score patterns. Finally, we provide...

Figures (3)

• Figure 1: Overview of different LLM-as-a-judge methods.
• Figure 2: Score distributions of selected metrics. $\mu,\sigma^2$ refer to the means and variances of scores for code translation, code generation, and code summarization. All scores are rescaled into range $[0, 1]$.
• Figure 3: Case study. The successful case from code translation is on the left while the failing case from code summarization is on the right.