Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation

Jinhao Dong, Jun Sun, Wenjie Zhang, Jin Song Dong, Dan Hao

TL;DR

ConAIR is an approach that aims to improve the performance of a code generator through two distinctive ingredients, i.e., lightweight user effort for validating the correctness of selected tests and a dynamic strategy for ranking, localizing and correcting multiple tests and codes.

Abstract

Code generation techniques generate code snippets automatically based on the problem requirements in natural language. Recently, large language models (LLMs) achieve the SOTA performance on code generation. However, LLMs still struggle at times to generate accurate code, which diminishes their promised efficiency as developers must spend significant effort evaluating and debugging the generated code. To improve the reliability and quality of the generated codes, researchers propose to leverage Consistency to obtain a better code based on generating and ranking multiple candidates. The existing approach is problematic as Consistency thinks a code is better when (1) the code pass more tests (inter-consistency) (2) more codes share the same behavior (intra-consistency). However, because the tests are also generated by LLMs, they could be wrong as well. As a result, majority voting based on testing results is unreliable. Relying solely on consistency is insufficient to address this issue; integrating user feedback is essential for effectively guiding consistency. We show that with minimal human effort, performance can be significantly enhanced. We propose Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation, ConAIR, which is an approach that aims to improve the performance of a code generator through two distinctive ingredients, i.e., (1) lightweight user effort for validating the correctness of selected tests; and (2) a dynamic strategy for ranking, localizing and correcting multiple tests and codes. Overall, we propose a lightweight interaction framework that incorporates user feedback to correct identified tests and guide the iterative process. The iteration rounds are only 4 in average with the help of consistency. With only lightweight human efforts, we can achieve an improvement of 33% towards the base model.

ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation

TL;DR

ConAIR is an approach that aims to improve the performance of a code generator through two distinctive ingredients, i.e., lightweight user effort for validating the correctness of selected tests and a dynamic strategy for ranking, localizing and correcting multiple tests and codes.

Abstract

Code generation techniques generate code snippets automatically based on the problem requirements in natural language. Recently, large language models (LLMs) achieve the SOTA performance on code generation. However, LLMs still struggle at times to generate accurate code, which diminishes their promised efficiency as developers must spend significant effort evaluating and debugging the generated code. To improve the reliability and quality of the generated codes, researchers propose to leverage Consistency to obtain a better code based on generating and ranking multiple candidates. The existing approach is problematic as Consistency thinks a code is better when (1) the code pass more tests (inter-consistency) (2) more codes share the same behavior (intra-consistency). However, because the tests are also generated by LLMs, they could be wrong as well. As a result, majority voting based on testing results is unreliable. Relying solely on consistency is insufficient to address this issue; integrating user feedback is essential for effectively guiding consistency. We show that with minimal human effort, performance can be significantly enhanced. We propose Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation, ConAIR, which is an approach that aims to improve the performance of a code generator through two distinctive ingredients, i.e., (1) lightweight user effort for validating the correctness of selected tests; and (2) a dynamic strategy for ranking, localizing and correcting multiple tests and codes. Overall, we propose a lightweight interaction framework that incorporates user feedback to correct identified tests and guide the iterative process. The iteration rounds are only 4 in average with the help of consistency. With only lightweight human efforts, we can achieve an improvement of 33% towards the base model.

Paper Structure

This paper contains 30 sections, 2 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Effectiveness of Consistency and Limitations of Relying Solely on It
  4. Consequences of Neglecting Consistency Preconditions
  5. Approach
  6. Task Definition
  7. Lightweight Interaction Framework to Gather User Feedback
  8. Co-Evolution Process between Codes and Tests
  9. Ranking
  10. Correcting.
  11. Fixing.
  12. Experimental Setup
  13. Research Questions
  14. Dataset
  15. Evaluation Metrics
  16. ...and 15 more sections

Figures (5)

  • Figure 1: ConAIR built on GPT-3.5 surpasses the SOTA general LLM GPT-4o and the SOTA post-processing technique MPSC on all datasets.
  • Figure 2: The motivating example for limitations of relying solely on consistency (HumanEval/135)
  • Figure 3: The motivating example illustrating the consequences of ignoring the prerequisite required for consistency (HumanEval/41)
  • Figure 4: The overview of ConAIR
  • Figure 5: The co-evolution process of ConAIR