Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation

Thai Tang Quoc, Duc Ha Minh, Tho Quan Thanh, Anh Nguyen-Duc

TL;DR

CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process, guided by a chain of thought constructed from real-world programming problem feedback, providing a practical solution for improving LLM-based code generation.

Abstract

Large Language Models (LLMs) have recently advanced many applications on software engineering tasks, particularly the potential for code generation. Among contemporary challenges, code generated by LLMs often suffers from inaccuracies and hallucinations, requiring external inputs to correct. One recent strategy to fix these issues is to refine the code generated from LLMs using the input from the model itself (self-augmented). In this work, we proposed a novel method, namely CoT-SelfEvolve. CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process, guided by a chain of thought constructed from real-world programming problem feedback. Focusing on data science code, including Python libraries such as NumPy and Pandas, our evaluations on the DS-1000 dataset demonstrate that CoT-SelfEvolve significantly outperforms existing models in solving complex problems. The framework shows substantial improvements in both initial code generation and subsequent iterations, with the model's accuracy increasing significantly with each additional iteration. This highlights the effectiveness of using chain-of-thought prompting to address complexities revealed by program executor traceback error messages. We also discuss how CoT-SelfEvolve can be integrated into continuous software engineering environments, providing a practical solution for improving LLM-based code generation.

An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation

TL;DR

CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process, guided by a chain of thought constructed from real-world programming problem feedback, providing a practical solution for improving LLM-based code generation.

Abstract

Large Language Models (LLMs) have recently advanced many applications on software engineering tasks, particularly the potential for code generation. Among contemporary challenges, code generated by LLMs often suffers from inaccuracies and hallucinations, requiring external inputs to correct. One recent strategy to fix these issues is to refine the code generated from LLMs using the input from the model itself (self-augmented). In this work, we proposed a novel method, namely CoT-SelfEvolve. CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process, guided by a chain of thought constructed from real-world programming problem feedback. Focusing on data science code, including Python libraries such as NumPy and Pandas, our evaluations on the DS-1000 dataset demonstrate that CoT-SelfEvolve significantly outperforms existing models in solving complex problems. The framework shows substantial improvements in both initial code generation and subsequent iterations, with the model's accuracy increasing significantly with each additional iteration. This highlights the effectiveness of using chain-of-thought prompting to address complexities revealed by program executor traceback error messages. We also discuss how CoT-SelfEvolve can be integrated into continuous software engineering environments, providing a practical solution for improving LLM-based code generation.
Paper Structure (30 sections, 7 figures, 3 tables)

This paper contains 30 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs and Automated Program Repair
  4. Improving the performance of LLMs
  5. Via Learning from Human Feedback
  6. Via Learning with Automated Feedback
  7. Self-correcting LLMs
  8. Self-correcting LLMs in APR
  9. Proposed approach
  10. Evaluations
  11. Experimental settings
  12. Benchmark data
  13. External Knowledge Base
  14. Evaluation Metrics
  15. Results
  16. ...and 15 more sections

Figures (7)

  • Figure 1: The architecture of CoT-SelfEvolve framework
  • Figure 2: (RQ1) Comparing performance results for SelfEvolve and CoT-SelfEvolve across different libraries.
  • Figure 3: (RQ1) DS-1000 average performance across various LLMs. (%)
  • Figure 4: (RQ3) Cumulative number of problems reaching the stop condition at different attempts where $\text{n=5}$.
  • Figure 5: (RQ3) Number of prompt tokens and completion tokens for different max attempts $n$.
  • ...and 2 more figures