Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation

Atharva Naik

TL;DR

This study critically evaluates embedding-based evaluation of NL2Code systems, focusing on CodeBERTScore as a proxy for functional correctness. Using HumanEval and MBPP across ten diverse models, it shows only a weak link to actual execution success while strongly reflecting editing effort, challenging the use of embedding-based metrics as functional correctness surrogates. The work demonstrates robustness to perturbations but limited discriminative power for correctness and advocates pursuing reference-free or execution-based evaluation methods (e.g., CodeScore, RealHumanEval) to better capture real-world coding productivity and correctness. Overall, embedding-based metrics appear more suitable for measuring syntactic similarity and editing effort than true functional equivalence in generated code.

Abstract

The task of code generation from natural language (NL2Code) has become extremely popular, especially with the advent of Large Language Models (LLMs). However, efforts to quantify and track this progress have suffered due to a lack of reliable metrics for functional correctness. While popular benchmarks like HumanEval have test cases to enable reliable evaluation of correctness, it is time-consuming and requires human effort to collect test cases. As an alternative several reference-based evaluation metrics have been proposed, with embedding-based metrics like CodeBERTScore being touted as having a high correlation with human preferences and functional correctness. In our work, we analyze the ability of embedding-based metrics like CodeBERTScore to measure functional correctness and other helpful constructs like editing effort by analyzing outputs of ten models over two popular code generation benchmarks. Our results show that while they have a weak correlation with functional correctness (0.16), they are strongly correlated (0.72) with editing effort.

On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation

TL;DR

This study critically evaluates embedding-based evaluation of NL2Code systems, focusing on CodeBERTScore as a proxy for functional correctness. Using HumanEval and MBPP across ten diverse models, it shows only a weak link to actual execution success while strongly reflecting editing effort, challenging the use of embedding-based metrics as functional correctness surrogates. The work demonstrates robustness to perturbations but limited discriminative power for correctness and advocates pursuing reference-free or execution-based evaluation methods (e.g., CodeScore, RealHumanEval) to better capture real-world coding productivity and correctness. Overall, embedding-based metrics appear more suitable for measuring syntactic similarity and editing effort than true functional equivalence in generated code.

Abstract

The task of code generation from natural language (NL2Code) has become extremely popular, especially with the advent of Large Language Models (LLMs). However, efforts to quantify and track this progress have suffered due to a lack of reliable metrics for functional correctness. While popular benchmarks like HumanEval have test cases to enable reliable evaluation of correctness, it is time-consuming and requires human effort to collect test cases. As an alternative several reference-based evaluation metrics have been proposed, with embedding-based metrics like CodeBERTScore being touted as having a high correlation with human preferences and functional correctness. In our work, we analyze the ability of embedding-based metrics like CodeBERTScore to measure functional correctness and other helpful constructs like editing effort by analyzing outputs of ten models over two popular code generation benchmarks. Our results show that while they have a weak correlation with functional correctness (0.16), they are strongly correlated (0.72) with editing effort.
Paper Structure (18 sections, 17 equations, 4 figures, 6 tables)

This paper contains 18 sections, 17 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Related Work
  4. Datasets
  5. Metrics
  6. Methods
  7. Datasets
  8. Systems
  9. Metric Properties
  10. Boundedness
  11. Score Distribution
  12. Number of ties
  13. Convergent and Discriminant Validity
  14. Discriminative Power
  15. Empirical Analysis
  16. ...and 3 more sections

Figures (4)

  • Figure 1: The distribution of metric values. The x-axis shows the range of values while the y-axis shows the number of model decisions with values in that range. The distributions clearly show that the embedding metrics tend to over-reward decisions while the other metrics tend to over-penalize them. The chrF metric has the least skew
  • Figure 2: Correlations of the CodeBERTScore metrics with the other metrics. Except for exact match, most of the metrics have a moderate correlation with CodeBERTScore
  • Figure 3: The achieved significance level of the metric values. x-axis shows pairs of runs while y-axis shows the p-values. The dotted line shows the acceptance threshold
  • Figure 4: The achieved significance level of the metric values. x-axis shows pairs of runs while y-axis shows the p-values. The dotted line shows the acceptance threshold. All p-values above $0.008$ are clipped to zoom into the portion close to the acceptance threshold.