Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

Simin Chen, Xiaoning Feng, Xiaohong Han, Cong Liu, Wei Yang

TL;DR

The paper addresses the challenge of benchmarking large code generation models by automatically generating diverse programming problems. It introduces Programming Problem Merging (PPM), which creates new problems by merging seed tasks and applying metamorphic transformations guided by return-value analysis, with Lambda Programming Problems ensuring natural descriptions. Two implementations, PPM-T and PPM-V, along with eight LCGMs and nine baselines, are evaluated on HumanEval and MBPP, demonstrating significantly increased semantic diversity, more natural prompts, and improved data integrity, while substantially reducing model correctness on the generated benchmarks. The results show that PPM provides stable benchmarking across random trials and hyperparameter settings, offering a practical, replication-friendly approach to robustly evaluate code-generation capabilities in real-world scenarios.

Abstract

In recent times, a plethora of Large Code Generation Models (LCGMs) have been proposed, showcasing significant potential in assisting developers with complex programming tasks. Benchmarking LCGMs necessitates the creation of a set of diverse programming problems, and each problem comprises the prompt (including the task description), canonical solution, and test inputs. The existing methods for constructing such a problem set can be categorized into two main types: manual methods and perturbation-based methods. However, manual methods demand high effort and lack scalability, while also risking data integrity due to LCGMs' potentially contaminated data collection, and perturbation-based approaches mainly generate semantically homogeneous problems with the same canonical solutions and introduce typos that can be easily auto-corrected by IDE, making them ineffective and unrealistic. In this work, we propose the idea of programming problem merging (PPM) and provide two implementation of this idea, we utilize our tool on two widely-used datasets and compare it against nine baseline methods using eight code generation models. The results demonstrate the effectiveness of our tool in generating more challenging, diverse, and natural programming problems, comparing to the baselines.

PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

TL;DR

The paper addresses the challenge of benchmarking large code generation models by automatically generating diverse programming problems. It introduces Programming Problem Merging (PPM), which creates new problems by merging seed tasks and applying metamorphic transformations guided by return-value analysis, with Lambda Programming Problems ensuring natural descriptions. Two implementations, PPM-T and PPM-V, along with eight LCGMs and nine baselines, are evaluated on HumanEval and MBPP, demonstrating significantly increased semantic diversity, more natural prompts, and improved data integrity, while substantially reducing model correctness on the generated benchmarks. The results show that PPM provides stable benchmarking across random trials and hyperparameter settings, offering a practical, replication-friendly approach to robustly evaluate code-generation capabilities in real-world scenarios.

Abstract

In recent times, a plethora of Large Code Generation Models (LCGMs) have been proposed, showcasing significant potential in assisting developers with complex programming tasks. Benchmarking LCGMs necessitates the creation of a set of diverse programming problems, and each problem comprises the prompt (including the task description), canonical solution, and test inputs. The existing methods for constructing such a problem set can be categorized into two main types: manual methods and perturbation-based methods. However, manual methods demand high effort and lack scalability, while also risking data integrity due to LCGMs' potentially contaminated data collection, and perturbation-based approaches mainly generate semantically homogeneous problems with the same canonical solutions and introduce typos that can be easily auto-corrected by IDE, making them ineffective and unrealistic. In this work, we propose the idea of programming problem merging (PPM) and provide two implementation of this idea, we utilize our tool on two widely-used datasets and compare it against nine baseline methods using eight code generation models. The results demonstrate the effectiveness of our tool in generating more challenging, diverse, and natural programming problems, comparing to the baselines.
Paper Structure (28 sections, 4 equations, 6 figures, 7 tables, 1 algorithm)

This paper contains 28 sections, 4 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Large Code Generation Models
  4. Programming Problem Generation Methods
  5. Challenges & High-level Solutions
  6. Approach
  7. Problem Formulation of Programming Problem Merging
  8. Design Overview
  9. Return Value Type Analysis
  10. Metamorphic Transformation Operator Selection
  11. Prompt and Canonical Solution Construction
  12. Benchmarking Stage
  13. Experimental Setup
  14. Datasets
  15. Pre-trained Code Generation Models
  16. ...and 13 more sections

Figures (6)

  • Figure 1: Programming problem example
  • Figure 2: Design overview of PPM
  • Figure 3: DiffImp of PPM
  • Figure 4: Naturalness results
  • Figure 5: The effectiveness of PPM with different numbers of generated candidates (n)
  • ...and 1 more figures