MATCH: Task-Driven Code Evaluation through Contrastive Learning

TL;DR

MATCH tackles the challenge of evaluating NL-to-code outputs without reference solutions or executions by learning cross-modal embeddings that align natural language task descriptions with code snippets. It employs an Enhanced Embeddings Layer with either Cross-Attention or Linear augmentation to produce task and code representations and uses cosine similarity to derive the score $f(t,c)$, with $sim(t,c)=\frac{1+f(t,c)}{2}$ for continuous signals. The model is trained via binary and continuous loss functions, $\mathcal{L}_{bin}$ and $\mathcal{L}_{cont}$, to reflect functional correctness and human preferences, respectively. Across HumanEval, MBPP-Eval, and CoNaLa benchmarks in Java, Python, JavaScript, and C++, MATCH achieves stronger correlations with functional correctness and human judgments than CodeBERTScore, ICE-Score, and CodeScore, demonstrating practical, reference-free evaluation for NL-to-code generation. This approach offers a scalable, versatile tool for code evaluation in settings where reference code or exhaustive testing is unavailable, with potential extensions to non-functional quality dimensions.

Abstract

AI-based code generation is increasingly prevalent, with GitHub Copilot estimated to generate 46% of the code on GitHub. Accurately evaluating how well generated code aligns with developer intent remains a critical challenge. Traditional evaluation methods, such as unit tests, are often unscalable and costly. Syntactic similarity metrics (e.g., BLEU, ROUGE) fail to capture code functionality, and metrics like CodeBERTScore require reference code, which is not always available. To address the gap in reference-free evaluation, with few alternatives such as ICE-Score, this paper introduces MATCH, a novel reference-free metric. MATCH uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions, enabling similarity scoring that reflects how well generated code implements the task. We show that MATCH achieves stronger correlations with functional correctness and human preference than existing metrics across multiple programming languages.

Figures (3)

• Figure 1: Two examples from the HumanEval dataset that illustrate MATCH's evaluation ability for correct and incorrect code snippets given a description (for MATCH) and a reference code (for CodeBERTScore, denoted as CBS). The type of incorrectness for the incorrect code is described as well. We show the score for both approaches for both correct and incorrect Python code, displaying MATCH's ability to differentiate between them compared to CodeBERTScore.
• Figure 2: MATCH Architecture: This figure illustrates the general architecture of MATCH, featuring an Enhanced Embeddings Layer that integrates contextual information from task descriptions and code snippets. Specific implementations, including Linear and Cross-Attention enhancements, are shown in \ref{['fig:enhancement_layers']}. The architecture computes similarity between learned embeddings using an appropriate similarity function. such as Cosine Similarity.
• Figure 3: An illustration of the specific implementations of the Enhanced Embeddings Layer: the Cross-Attention and Linear layers.