Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach

Longtian Wang, Tianlin Li, Xiaofei Xie, Yuhan Zhi, Jian Wang, Chao Shen

TL;DR

This work critiques the reliance on single-prompt benchmarks for code-generation assessment and introduces a mutation-based framework to simulate real-world prompt variations. By generating 12,834 prompt variants through 10 mutation strategies and evaluating five Code Large Language Models, it reveals substantial gaps between traditional benchmark results and performance under diverse descriptions. The study introduces three metrics—Correctness Variability, Mutation Bias, and Best Pass@k—to quantify robustness to prompt perturbations and demonstrates that existing benchmarks can bias model rankings. The findings advocate more robust, diverse evaluation designs to fairly compare CLLMs and guide future benchmark development for practical code-generation tasks.

Abstract

Code Large Language Models (CLLMs) have exhibited outstanding performance in program synthesis, attracting the focus of the research community. The evaluation of CLLM's program synthesis capability has generally relied on manually curated benchmarks. However, there is a substantial gap between real-world scenarios and benchmark settings. Existing benchmarks typically provide only a single input prompt for the evaluation of each synthesis problem. However, in practice, a problem can be described in various ways, including with typos, where developers may struggle to understand certain descriptions and seek clarification to find more suitable wording. Such various descriptions may lead to variations in the performance of CLLMs on the same question, resulting in a biased evaluation when using existing benchmarks. In this paper, we aim to explore these pitfalls with the goal of revisiting and enhancing future benchmark designs. To simulate real-world variations in problem descriptions, we propose 10 mutation strategies and introduce three new metrics to evaluate their impact on code generation. We then assess five popular CLLMs using 12,834 generated prompt variants, and found a significant performance discrepancy between the results from existing benchmarks and those from mutated benchmarks containing perturbations and variations. This finding underscores the need for more robust evaluation methods and benchmarks.

Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach

TL;DR

This work critiques the reliance on single-prompt benchmarks for code-generation assessment and introduces a mutation-based framework to simulate real-world prompt variations. By generating 12,834 prompt variants through 10 mutation strategies and evaluating five Code Large Language Models, it reveals substantial gaps between traditional benchmark results and performance under diverse descriptions. The study introduces three metrics—Correctness Variability, Mutation Bias, and Best Pass@k—to quantify robustness to prompt perturbations and demonstrates that existing benchmarks can bias model rankings. The findings advocate more robust, diverse evaluation designs to fairly compare CLLMs and guide future benchmark development for practical code-generation tasks.

Abstract

Code Large Language Models (CLLMs) have exhibited outstanding performance in program synthesis, attracting the focus of the research community. The evaluation of CLLM's program synthesis capability has generally relied on manually curated benchmarks. However, there is a substantial gap between real-world scenarios and benchmark settings. Existing benchmarks typically provide only a single input prompt for the evaluation of each synthesis problem. However, in practice, a problem can be described in various ways, including with typos, where developers may struggle to understand certain descriptions and seek clarification to find more suitable wording. Such various descriptions may lead to variations in the performance of CLLMs on the same question, resulting in a biased evaluation when using existing benchmarks. In this paper, we aim to explore these pitfalls with the goal of revisiting and enhancing future benchmark designs. To simulate real-world variations in problem descriptions, we propose 10 mutation strategies and introduce three new metrics to evaluate their impact on code generation. We then assess five popular CLLMs using 12,834 generated prompt variants, and found a significant performance discrepancy between the results from existing benchmarks and those from mutated benchmarks containing perturbations and variations. This finding underscores the need for more robust evaluation methods and benchmarks.
Paper Structure (28 sections, 1 equation, 4 figures, 2 tables)

This paper contains 28 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Code Large Language Model
  4. CLLM Evaluation
  5. Preliminary
  6. CLLM Benchmark
  7. Pass@k metric
  8. Methodology
  9. Overview
  10. Mutation Strategy
  11. Typo Simulation
  12. Synonym Substitution
  13. LLM Paraphrase
  14. LLM Summarize
  15. Example Insertion and Removal
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Two input prompts for the same synthesis problem. The difference between the two prompts is highlighted. The correctness of generated code can be significantly improved by making slight adjustments
  • Figure 2: Workflow of our evaluation
  • Figure 3: The effects of all mutation strategies, the modifications to the input prompt after mutation are highlighted
  • Figure 4: MB across Mutation Strategies