Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Creative and Correct: Requesting Diverse Code Solutions from AI Foundation Models

Scott Blyth, Markus Wagner, Christoph Treude

TL;DR

This study systematically investigates the trade-off between diversity and correctness in AI foundation models using experiments with HumanEval tasks, identifying combinations of parameters and strategies that strike an optimal balance between diversity and correctness.

Abstract

AI foundation models have the capability to produce a wide array of responses to a single prompt, a feature that is highly beneficial in software engineering to generate diverse code solutions. However, this advantage introduces a significant trade-off between diversity and correctness. In software engineering tasks, diversity is key to exploring design spaces and fostering creativity, but the practical value of these solutions is heavily dependent on their correctness. Our study systematically investigates this trade-off using experiments with HumanEval tasks, exploring various parameter settings and prompting strategies. We assess the diversity of code solutions using similarity metrics from the code clone community. The study identifies combinations of parameters and strategies that strike an optimal balance between diversity and correctness, situated on the Pareto front of this trade-off space. These findings offer valuable insights for software engineers on how to effectively use AI foundation models to generate code solutions that are diverse and accurate.

Creative and Correct: Requesting Diverse Code Solutions from AI Foundation Models

TL;DR

This study systematically investigates the trade-off between diversity and correctness in AI foundation models using experiments with HumanEval tasks, identifying combinations of parameters and strategies that strike an optimal balance between diversity and correctness.

Abstract

AI foundation models have the capability to produce a wide array of responses to a single prompt, a feature that is highly beneficial in software engineering to generate diverse code solutions. However, this advantage introduces a significant trade-off between diversity and correctness. In software engineering tasks, diversity is key to exploring design spaces and fostering creativity, but the practical value of these solutions is heavily dependent on their correctness. Our study systematically investigates this trade-off using experiments with HumanEval tasks, exploring various parameter settings and prompting strategies. We assess the diversity of code solutions using similarity metrics from the code clone community. The study identifies combinations of parameters and strategies that strike an optimal balance between diversity and correctness, situated on the Pareto front of this trade-off space. These findings offer valuable insights for software engineers on how to effectively use AI foundation models to generate code solutions that are diverse and accurate.
Paper Structure (20 sections, 2 equations, 2 figures, 2 tables)

This paper contains 20 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. HumanEval Tasks
  3. Diversity Assessment
  4. Correctness Assessment
  5. Configuring Foundation Models
  6. temperature
  7. top_p
  8. frequency_penalty
  9. presence_penalty
  10. logit bias
  11. Prompting Techniques
  12. Regeneration
  13. n_different
  14. n_k_different
  15. Prompt Classification
  16. ...and 5 more sections

Figures (2)

  • Figure 1: n_k_different prompt with Logit Bias; UML sequence diagram.
  • Figure 2: Code similarity and correctness for all 20 approaches from Section \ref{['sec:oneatatime']} and for the 2 approaches from Section \ref{['sec:combination']}. Mean similarity scores are reported across all 164 HumanEval tasks. The red star shows the starting point of our investigations (A0). Sub-figures (a) and (b): the purple/green triangles represent the Pareto fronts (from left to right): purple A15, A14, A1, A20, green A15, A1, A20. The light-green diamonds represent A21 and A22 in Section \ref{['sec:combination']}. Sub-figure (c): we show the correlation of the two clone detection approaches (Spearman correlation coefficient 0.993).