Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

TL;DR

The paper investigates how offline metrics for code-generation align with real-world programmer value in human-AI pair programming. It shows that while unit-test-based correctness strongly signals value, many useful generations fail tests, and that syntactic similarity offers complementary information. A simple Combined metric blending Pass with Edit-Sim improves correlation with perceived value by about 14%, suggesting a practical path to better evaluate and compare code-generation systems. The findings advocate multidimensional, human-centered evaluation and provide concrete guidance for deploying evaluation metrics in development and decision-making contexts.

Abstract

Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code is most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.

Figures (4)

• Figure 1: In the example above (counting even and odd numbers), code suggested by a model fails unit tests but is deemed useful by programmers because adding a short check ($abs$ value) fixes the generation.
• Figure 2: Correlation (Pearson) between human judgements (perceived value, accuracy and effort) and offline metrics (functional correctness, edit similarity and a combined metric (see section \ref{['sec:combinedmetric']})). All correlations are significant with $p < 0.001$.
• Figure 3: Distributions of Value judgments, in cases where generations pass unit tests ($\textsc{Pass}\xspace=1$, green), and fail unit tests (Pass=0, blue). When generations pass unit tests, they are likely to be judged as valuable ($\textsc{Value}\xspace \geq 3$). In fact, only 3% of generations fall below this score. When functions fail unit tests, they are somewhat more likely to have lower Value scores, but 90 generations (42%) are still considered "somewhat useful" by participants. The metric misses many high-value generations.
• Figure 4: Distributions of Value judgments, in the four possible combinations of Pass outcomes and low/high Edit-Sim scores. The top and bottom rows indicate cases where Edit-Sim falls below and above the 50th-percentile, respectively. The left and right columns indicate cases where $\textsc{Pass}\xspace=0$ and $\textsc{Pass}\xspace=1$ respectively. When $\textsc{Pass}\xspace=1$, generations are likely be high value (blue regions). When both $\textsc{Pass}\xspace=0$ and Edit-Sim=low, generations are likely to be low value (red region). The Value is more uniformly distributed in the remaining region where both $\textsc{Pass}\xspace=0$ and Edit-Sim=high.