Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes

Heejae Chon, Seonghyeon Lee, Jinyoung Yeo, Dongha Lee

TL;DR

The paper argues that evaluating code-language models should extend beyond functional correctness to include diversity of generated solutions. It proposes a unified framework with three pairwise similarity measures (token-based, embedding-based, and reasoning-based) and two joint metrics, Sim@K and CSim@K, plus a diversity-weighted correctness metric DPass@K, to assess $K$ candidate codes. Through experiments on APPS and HumanEval across multiple models and prompting strategies, the authors show that current LMs tend to produce functionally correct but highly similar code; instruction-tuning increases similarity, while increasing diversity often requires higher temperatures or different prompting approaches. A reasoning-based similarity measure (LLM-Eval) best aligns with human judgments, underscoring the value of human-aligned, model-assisted evaluation for code generation and suggesting pathways to foster greater diversity without sacrificing correctness.

Abstract

Language models (LMs) have exhibited impressive abilities in generating codes from natural language requirements. In this work, we highlight the diversity of code generated by LMs as a critical criterion for evaluating their code generation capabilities, in addition to functional correctness. Despite its practical implications, there is a lack of studies focused on assessing the diversity of generated code, which overlooks its importance in the development of code LMs. We propose a systematic approach to evaluate the diversity of generated code, utilizing various metrics for inter-code similarity as well as functional correctness. Specifically, we introduce a pairwise code similarity measure that leverages large LMs' capabilities in code understanding and reasoning, demonstrating the highest correlation with human judgment. We extensively investigate the impact of various factors on the quality of generated code, including model sizes, temperatures, training approaches, prompting strategies, and the difficulty of input problems. Our consistent observation of a positive correlation between the test pass score and the inter-code similarity score indicates that current LMs tend to produce functionally correct code with limited diversity.

Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes

TL;DR

The paper argues that evaluating code-language models should extend beyond functional correctness to include diversity of generated solutions. It proposes a unified framework with three pairwise similarity measures (token-based, embedding-based, and reasoning-based) and two joint metrics, Sim@K and CSim@K, plus a diversity-weighted correctness metric DPass@K, to assess candidate codes. Through experiments on APPS and HumanEval across multiple models and prompting strategies, the authors show that current LMs tend to produce functionally correct but highly similar code; instruction-tuning increases similarity, while increasing diversity often requires higher temperatures or different prompting approaches. A reasoning-based similarity measure (LLM-Eval) best aligns with human judgments, underscoring the value of human-aligned, model-assisted evaluation for code generation and suggesting pathways to foster greater diversity without sacrificing correctness.

Abstract

Language models (LMs) have exhibited impressive abilities in generating codes from natural language requirements. In this work, we highlight the diversity of code generated by LMs as a critical criterion for evaluating their code generation capabilities, in addition to functional correctness. Despite its practical implications, there is a lack of studies focused on assessing the diversity of generated code, which overlooks its importance in the development of code LMs. We propose a systematic approach to evaluate the diversity of generated code, utilizing various metrics for inter-code similarity as well as functional correctness. Specifically, we introduce a pairwise code similarity measure that leverages large LMs' capabilities in code understanding and reasoning, demonstrating the highest correlation with human judgment. We extensively investigate the impact of various factors on the quality of generated code, including model sizes, temperatures, training approaches, prompting strategies, and the difficulty of input problems. Our consistent observation of a positive correlation between the test pass score and the inter-code similarity score indicates that current LMs tend to produce functionally correct code with limited diversity.
Paper Structure (26 sections, 3 equations, 6 figures, 8 tables)

This paper contains 26 sections, 3 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Generation with Language Models
  4. Similarity Measure between Codes
  5. Methods
  6. Pairwise Code Similarity Measurement
  7. Token-based similarity
  8. Embedding-based similarity
  9. Reasoning-based similarity
  10. Similarity Metric for $K$ Sampled Codes
  11. Joint Metric for $K$ Sampled Codes: Similarity and Functional Correctness
  12. Experiments
  13. Experimental Setup
  14. Datasets
  15. Models
  16. ...and 11 more sections

Figures (6)

  • Figure 1: In complex code generation tasks, utilizing diversity encoded in LMs help generate correct outputs.
  • Figure 2: The evaluation pipeline for an LM's code generation ability in terms of inter-code similarity (i.e., a negative indicator of diversity) while focusing on its $K$ generated codes, termed as Sim@$K$. We adopt three different approaches to pairwise code similarity evaluation, which are based on (1) token-based (2) embedding-based, and (3) reasoning-based similarity measures.
  • Figure 3: Code-related concepts and their hierarchy from a software engineering perspective.
  • Figure 4: Similarity and functional correctness of generated codes from base models (pentagons) and instruction-tuned models (stars) on HumanEval.
  • Figure 5: Jointly evaluation of functional correctness and similarity score on the HumanEval dataset.
  • ...and 1 more figures